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EXECUTIVE  SUMMARY 


Introduction  to  the  research  and  development  project 

The  assessment  of  mariner  proficiencies  by  practical  demonstration  is  mandated  by  the 
International  Maritime  Organization  (IMO)  in  its  1995  amendments  to  the  International 
Convention  on  Standards  of  Training,  Certification  and  Watchkeeping  for  Seafarers  (STCW 
Convention)  and  the  accompanying  Seafarer’s  Training,  Certification  and  Watchkeeping  Code 
(STCW  Code).  This  requirement  is  a  substantial  departure  from  earlier  practice,  and  methods  for 
developing,  conducting,  and  documenting  such  assessments  must  be  established.  The  United 
States  Coast  Guard’s  (USCG)  National  Maritime  Center  (NMC)  is  sponsoring  a  research  project 
through  the  USCG  Research  and  Development  Center  (R&DC)  to  examine  the  implications  of 
the  mandate  and  to  ensure  that  the  best  practices  are  available  to  the  industry.  One  major 
objective  of  the  project  is  to  provide  a  systematic,  step-by-step  process  for  developing  reliable 
and  valid  assessments  and  investigate  its  feasibility  for  implementation. 

Project  efforts  reported  here 

The  performance-based  assessment  (PBA)  method,  as  described  in  this  report,  is  compliant  with 
the  requirements  of  the  STCW  Code,  relevant  USCG  Navigation  and  Vessel  Inspection  Circulars 
(NVICs),  Instructional  Systems  Development  (ISD),  and  the  best  practices  of  industry.  An 
objective  of  this  phase  of  the  project  was  to  refine  the  PBA  method  and  to  prepare  materials  to 
assist  the  industry  in  applying  it.  The  approach  to  achieving  this  objective  was  to  introduce  the 
method  to  a  varied  sample  of  industry  representatives  through  workshops  and  to  invite  some  of 
them  to  participate  in  trial  applications  of  the  method.  As  a  result  of  these  efforts,  the  materials 
provided  in  this  report  have  benefited  from  multiple  reviews,  multiple  trial  applications,  and 
substantial  comment.  Materials  to  guide  the  development  of  assessment  procedures  that  are 
provided  here  include  a  workshop  (Chapter  2),  a  manual  (Chapter  3),  and  four  sample 
assessments  (Appendices  A  through  D).  The  industry  representatives  who  participated  in  the 
project  and  came  to  understand  the  method  were  quite  positive  about  the  value  of  the  general 
method  in  enhancing  the  rigor  of  assessment  procedures.  They  are  applying  it  in  their  own 
settings  -  sometimes  with  their  own  adaptations  to  their  particular  circumstances.  Comparable 
guidance  and  materials  to  support  the  conduct  of  assessments,  are  provided  in  a  separate  report 
(R&DC-204). 

Recommendations 

For  the  most  effective  implementation  of  the  project’s  findings  and  products,  the  following 
actions  are  recommended  to  the  USCG  and  the  maritime  industry. 

-  The  USCG  should  encourage  the  maritime  industry  in  the  use  of  the  PBA  method,  as 
described  in  the  project  reports  and  materials.  As  a  first  step,  the  materials  should  be 
made  widely  available,  on  the  USCG  STCW  website  (http://www.uscg.mil/stcw)  and 
through  the  National  Technical  Information  Service  (NTIS).  When  the  industry  becomes 
more  familiar  with  the  PBA  method,  it  can  serve  as  the  basis  for  a  new  NVIC.  The 
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USCG  should  submit  the  PBA  supporting  materials  to  the  IMO  subcommittee  on 
Standards  for  Training  and  Watchkeeping. 

•  The  USCG  should  encourage  further  familiarity  with  the  PBA  approach  by  USCG  staff, 
as  a  tool  for  discussions  of  assessment  with  the  industry  and  for  the  review  of  procedures 
submitted  for  approval. 

•  The  USCG  should  encourage  the  review  and  use  of  the  PBA  method  and  materials  by 
those  groups  that  are  dealing  with  the  important  technical  issues  of  assessment,  such  as 
the  Maritime  Academy  Simulator  Committee  (MASC),  the  Merchant  Marine  Personnel 
Advisory  Committee  (MERPAC),  and  academy  committees  appointed  by  the  Maritime 
Administration  (MARAD)  to  address  STCW  issues.  The  materials  can  provide  a 
common  approach  and  a  common  basis  for  discussion. 

■  Those  in  the  industry  who  are  responsible  for  training  and  assessment  of  mariner 

proficiency  in  academies,  training  schools,  and  shipping  companies  should  make  use  of 
the  PBA  method  and  the  materials  presented  here  as  a  guide  for  their  own  development 
of  assessment  procedures.  The  materials  are  especially  appropriate  for  inclusion  in  train- 
the-trainer  courses. 
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Chapter  1 

OVERVIEW 


This  chapter  provides  a  brief  overview  of  a  systematic  method  for 
developing  reliable  and  valid  performance-based  assessments.  It 
also  describes  other  materials  included  in  this  report  to  guide  the 
development  of  assessment  procedures:  a  workshop  (Chapter  2),  a 
developer’s  manual  (Chapter  3),  and  four  sample  assessments 
(Appendices  A  through  D). 
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web  site  at  http://www.rdc.uscg.mil. 
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INTRODUCTION 


The  assessment  of  mariner  proficiencies  by  practical  demonstration  is  mandated  for  numerous 
areas  of  competence  by  the  International  Maritime  Organization  (IMO)  in  its  1995  amendments 
to  the  International  Convention  on  Standards  of  Training,  Certification  and  Watchkeeping  for 
Seafarers  (STCW Convention)  (1996A)  and  the  accompanying  Seafarer’s  Training,  Certification 
and  Watchkeeping  Code  (STCW  Code).  In  response  to  this  requirement,  the  United  States  Coast 
Guard  (USCG)  has  provided  guidance  in  the  assessment  of  mariner  skills  and  knowledge  that 
calls  for  a  practical  demonstration  of  mariner  proficiency  (46  CFR  10.103;  46  CFR  12.01-6; 
USCG,  1997A;  USCG  1997B;  USCG  1997C).  Here, practical  demonstration  refers  to  mariner 
performance  of  an  activity  that  is  determined  to  meet  a  specified  standard  of  proficiency  by  an 
assessor. 

This  requirement  for  the  practical  demonstration  of  proficiency  is  a  substantial  departure  from 
earlier  assessment  practice  in  the  maritime  community.  In  order  to  meet  this  new  requirement, 
methods  for  developing,  conducting,  and  documenting  such  assessments  must  be  established. 
The  USCG  National  Maritime  Center  (NMC)  is  sponsoring  a  research  project  through  the  USCG 
Research  and  Development  Center  (R&DC)  to  examine  the  implications  of  these  requirements 
and  to  ensure  that  the  best  practices  are  available  to  the  maritime  industry.  Past  project  efforts 
have  resulted  in  the  specification  of  a  performance-based  assessment  (PBA)  method  for 
developing  mariner  assessments.  This  method  was  initially  refined  during  a  trial  application  to 
Automatic  Radar  Plotting  Aid  (ARP A)  operator  assessment  (McCallum,  Forsythe,  Smith, 
Nunnenkamp,  &  Sandberg,  2000).  The  trial  application  required  extensive  guidance  and 
technical  support  by  assessment  development  experts.  Following  that  application,  it  was 
concluded  that  broader  adoption  of  the  method  by  the  maritime  community  would  require  further 
refinement  of  the  PBA  method,  as  well  as  development  of  aids  to  support  application  of  the 
method.  Specific  aids  selected  for  development  included  a  workshop  on  assessment 
development,  a  practical  guide  for  assessment  development,  and  a  set  of  sample  assessment 
procedures. 

Objectives  and  Approach 

The  general  objective  of  the  present  effort  is  to  further  refine  the  PBA  method  and  to  develop 
supporting  materials  that  will  aid  in  the  broader  application  and  implementation  of  the  PBA 
method  within  the  maritime  community.  The  more  specific  objective  of  this  effort  is  to  provide 
practical  support  to  the  assessment  developers  and  assessors  who  must  respond  to  these 
requirements  within  the  United  States.  Three  related  products  were  identified  for  development: 

•  A  Workshop  for  Assessment  Developers  to  introduce  the  PBA  method  to  individuals 
responsible  for  developing  assessment  procedures. 

•  A  Manual  for  Assessment  Developers  to  provide  a  reference  and  aid  during  assessment 
procedure  development. 

•  A  set  of  sample  assessment  procedures  to  serve  as  content  and  format  guides  for 
developing  assessments  in  accordance  with  the  PBA  method. 
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Refinement  of  the  PBA  method  and  preparation  of  the  supporting  materials  were  complementary 
and  concurrent  activities.  First,  the  workshop  was  drafted  and  presented  to  government  and 
industry  representatives  of  the  maritime  community  in  order  to  refine  both  the  general  PBA 
method  and  the  workshop  materials.  Next,  faculty  members  from  two  maritime  academies 
developed  four  sample  assessment  procedures.  During  assessment  development,  the  faculty 
participants  provided  suggestions  regarding  the  guidance  required  for  preparing  assessment 
procedures,  assessment  procedure  content,  and  assessment  procedure  format.  Then,  a  draft  of  the 
assessment  development  manual  was  prepared  and  reviewed  by  representatives  of  the  maritime 
community.  Finally,  based  on  the  comments  received  and  lessons  learned,  all  of  the  PBA 
materials  were  revised  to  incorporate  consistent  methods,  procedures,  and  language. 

Report  Organization 

Chapter  1  of  this  report  is  intended  to  serve  as  a  brief  introduction,  and  is  divided  into  three 
sections.  First,  a  brief  overview  of  the  PBA  method,  as  it  applies  to  assessment  procedure 
development,  is  provided.  Then,  the  products  of  this  project-/!  Workshop  for  Assessment 
Developers,  A  Manual  for  Assessment  Developers,  and  the  four  sample  assessment  procedures  - 
are  briefly  described.  The  final  section  of  this  chapter  presents  conclusions  regarding  the 
technical  value  of  these  products  and  issues  concerning  their  implementation,  as  well  as 
recommendations  regarding  future  actions  that  should  be  taken  to  foster  broad  implementation  of 
the  PBA  method. 

Following  this  chapter  are  the  products  of  this  project.  These  products  are  stand-alone  resources 
that  can  be  used  by  the  maritime  community  to  aid  in  the  implementation  of  the  PBA  method. 
Chapter  2  provides  the  A  Workshop  for  Assessment  Developers  materials.  Chapter  3  A  Manual 
for  Assessment  Developers  provides  a  step-by-step  method  for  developing  performance-based 
assessments,  and  contains  a  glossary  of  assessment-related  terms.  Appendices  A,  B,  C,  and  D 
provide  the  sample  assessments. 
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ASSESSMENT  DEVELOPMENT  METHOD  OVERVIEW 


A  review  of  the  STCWCode,  other  IMO  documents,  USCG  guidance  in  Navigation  and  Vessel 
Inspection  Circulars  (NVICs),  the  methods  of  Instructional  Systems  Development  (ISD),  and  the 
best  practices  of  the  maritime  industry  was  performed.  From  these  sources,  a  step-by-step 
performance-based  assessment  (PBA)  method  for  developing  mariner  assessments  was  defined. 
In  its  broader  form,  the  PBA  method  also  addresses  the  conduct  of  assessments.  However,  the 
current  report  is  limited  to  a  consideration  of  assessment  development.  The  conduct  of 
assessments  will  be  the  topic  of  a  future  report. 

Three  basic  assumptions  underlie  the  PBA  method.  The  first  assumption  is  that  an  expert 
assessor  will  conduct  assessments  by  observing  a  practical  demonstration  by  a  mariner. 
Generally,  an  expert  assessor  is  a  mariner  qualified  in  the  proficiencies  being  assessed  and 
trained  in  the  procedures  of  assessment.  In  the  United  States,  an  expert  assessor  will  be  qualified 
as  a  designated  examiner,  as  defined  in  NVIC  6-97  (USCG,  1997B). 

The  second  assumption  is  that  the  objective  of  conducting  assessments  is  to  obtain  valid  and 
reliable  determinations  of  mariner  proficiency.  A  valid  assessment  accurately  determines 
mariners’  proficiency  in  meeting  the  skill,  knowledge,  and  performance  requirements  of  a  job.  A 
reliable  assessment  results  in  consistent  proficiency  determinations  between  assessment 
applications. 

The  third  basic  assumption  is  that  assessment  validity  and  reliability  require  the  prior 
development  of  assessment  procedures,  which  include  specification  of  mariner  performance 
measures,  performance  standards,  and  proficiency  criteria.  A  performance  measure  is  the 
procedure  used  for  observing  and  recording  a  mariner  action.  A  performance  standard  is  the 
level  of  a  performance  measure  that  must  be  achieved  to  be  acceptable  for  assessment  purposes. 
Proficiency  criteria  are  the  scoring  procedures  applied  in  determining  the  proficiency  of  a 
candidate  on  the  basis  of  performance  measures  and  performance  standards. 

Figure  1  depicts  the  five-step  PBA  method  for  developing  assessments  of  mariner  proficiency. 
The  important  first  step,  on  which  the  method  depends,  is  to  specify  assessment  objectives. 

Using  such  sources  as  the  STCW  Code,  established  procedures,  and  expert  opinion,  the 
performance  elements  to  be  assessed  are  listed  and  defined.  Careful  analysis  of  objectives 
provides  a  basis  to  determine  assessment  methods,  specify  assessment  conditions,  and  develop 
proficiency  criteria.  Finally,  with  the  analytical  work  completed,  the  assessment  developer  can 
prepare  assessment  materials.  The  assessment  materials  are  a  relatively  self-contained  set  of 
procedures  and  instructions  that  can  be  circulated  for  expert  review  and,  ultimately,  provided  to 
the  assessor  who  will  conduct  the  assessment.  This  approach  is  compatible  with  that  described 
in  IMO  circular  853  (IMO,  1998). 
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Figure  1.  Five-step  method  for  developing  mariner  performance-based  assessments. 

The  background  and  basis  for  the  PBA  method,  as  it  applies  to  assessment  development,  is 
described  in  greater  detail  in  a  more  lengthy  technical  report  (McCallum,  Forsythe,  Smith, 
Nunnenkamp,  &  Sandberg,  2000).  An  extension  of  the  method  to  evaluating  the  capability  of 
simulators  to  support  mariner  assessment  is  documented  in  a  separate  technical  report  (Raby, 
Forsythe,  McCallum,  &  Smith,  2000).  Guidance  in  the  practical  application  of  the  PBA  method 
is  discussed  in  the  next  section,  which  introduces  A  Workshop  for  Assessment  Developers 
(Chapter  2),  A  Manual  for  Assessment  Developers  (Chapter  3),  and  the  sample  assessment 
procedures  (Appendices  A,  B,  C,  and  D). 

Early  in  the  research  project,  it  became  apparent  that  developing  and  conducting  assessments 
were  separate  functions,  likely  to  be  performed  by  different  individuals  with  different 
backgrounds  and  sometimes  even  by  different  organizations.  The  efforts  described  in  this  report 
focused  on  developing  assessments.  These  materials  were  prepared  with  the  assumption  that  the 
resulting  assessment  materials  would  be  provided  to  an  assessor  who  would  prepare  for  and 
conduct  the  actual  assessment. 

More  recent  project  efforts  focus  on  providing  guidance  and  practical  aids  to  the  assessor. 
Currently,  training  officers  from  a  major  commercial  shipping  company  are  adapting  the 
assessment  procedures  presented  in  this  report  to  their  own  operations  and  equipment.  These 
training  officers  will  assist  ship  officers  in  assessing  candidate  proficiency  onboard  their  vessels. 
One  implementation  issue  of  immediate  concern  to  them  has  been  the  appropriate  level  of  detail 
in  the  specification  of  assessment  objectives,  measures,  and  standards.  Another  issue  concerns 
the  consistency  between  the  company’s  established  operating  procedures  and  the  performance 
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standards  required  by  the  assessments.  Findings  relevant  to  these  and  other  implementation 
issues  will  be  provided  in  a  future  report  (McCallum,  Barnes,  Forsythe,  and  Smith,  2000),  along 
with  guidance  and  practical  aids  for  conducting  assessments. 
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PRACTICAL  AIDS  FOR  DEVELOPING  MARINER  ASSESSMENTS 


This  section  of  the  report  introduces  the  three  products  resulting  from  this  phase  of  the  project, 
which  have  been  prepared  to  aid  the  maritime  community  in  the  practical  application  of  the  PBA 
method.  Each  product  is  briefly  described,  along  with  a  discussion  of  the  process  of  its 
development,  any  pertinent  findings  during  development  of  the  product,  and  issues  concerning 
implementation  or  further  refinement  of  the  product. 

A  Workshop  for  Developing  Mariner  Assessments 

Following  initial  definition  of  the  PBA  method,  it  was  recognized  that  a  training  workshop  could 
serve  as  a  useful  means  of  introducing  this  method  to  people  in  the  maritime  community  who 
would  be  responsible  for  some  aspect  of  mariner  assessment.  A  workshop  was  drafted  and 
presented  to  individuals  from  the  government  and  private  sectors  of  the  maritime  industry  in  two 
sessions  in  March  and  April  of  1998.  A  Workshop  for  Assessment  Developers  was  initially 
developed  as  a  three-day  introduction  to  developing  and  conducting  assessments.  However,  it 
was  later  recognized  that  the  two  functions  of  developing  and  conducting  assessments  could  best 
be  addressed  in  separate  workshops  focusing  on  the  assessment  developer  and  assessor  roles. 
This  report  provides  the  workshop  that  focuses  on  assessment  development  (See  Chapter  2).  A 
separate  report  (R&DC-204)  provides  guidance  and  practical  aids  for  the  conduct  of 
assessments. 

In  its  current  format,  the  workshop  consists  of  eight  modules.  Each  module  consists  of  lecture 
slides,  and  five  of  the  modules  have  practical  exercises,  as  summarized  in  Table  1 .  The 
workshop  is  estimated  to  require  approximately  two  days.  Chapter  2  provides  the  course 
curriculum  outline,  the  set  of  slides  used  to  support  the  lectures  and  exercises,  and  the  exercise 
materials. 


Table  1.  Summary  of  workshop  module  lecture  slides  and  practical  exercises. 


Workshop 

Module 

Lecture  Slides 

Practical  Exercise 

1 

Introduction  to  Assessment  Development 
Process 

None 

2 

Validity  and  Reliability 

1. 

Understanding  Validity  and  Reliability 

3 

Step  1 :  Specify  Assessment  Objectives 

2. 

Specifying  Assessment  Objectives 

4 

Step  2:  Determine  Assessment  Methods 

3. 

Determining  Assessment  Methods 

5 

Step  3:  Specify  Assessment  Conditions 

4. 

Specifying  Assessment  Conditions 

6 

Step  4:  Develop  Performance  Criteria 

5. 

Developing  Performance  Measures 

6. 

Developing  Performance  Standards 

7 

Step  5:  Prepare  Assessment  Materials 

None 

8 

Summary/Overview 

None 
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Participants  in  the  1998  sessions  generally  viewed  the  workshop  as  a  useful  means  of  introducing 
a  systematic  method  for  developing  mariner  assessments.  They  also  identified  a  number  of 
issues  that  they  thought  would  affect  the  successful  implementation  of  the  workshop.  First,  it 
was  seen  as  important  that  the  full  range  of  those  involved  in  assessment  development  within  the 
maritime  community  have  an  opportunity  to  participate,  including  government  personnel  who 
regulate  mariner  assessment,  faculty  members  of  maritime  academies  and  schools,  and  shipping 
company  training  staff.  A  second  critical  issue  concerned  the  qualifications  of  the  workshop 
leaders,  who  should  have  a  broad  understanding  of  instructional  system  development,  general 
knowledge  of  mariner  assessment  policy,  and  familiarity  with  mariner  proficiency  requirements. 
Finally,  many  workshop  participants  viewed  the  practical  exercises  as  central  to  the  learning 
experience.  They  stressed  the  importance  of  allowing  adequate  time  to  complete  each  exercise 
and  to  have  the  workshop  leader  conduct  a  critical  review  of  the  exercise  results. 

A  Manual  on  Developing  Mariner  Assessments 

The  initial  set  of  workshop  materials  included  a  set  of  job  aids  that  were  intended  to  help 
assessment  developers  in  applying  the  PBA  method.  Early  reviewers  of  these  job  aids  noted  their 
strong  preference  for  an  integrated  manual  that  could  be  used  as  a  reference  and  aid  throughout 
the  assessment  development  process.  In  response,  A  Manual  for  Assessment  Developers  was 
prepared  (See  Chapter  3).  Early  drafts  of  this  manual  were  reviewed  by  several  participants  in 
the  current  project  and  refined  in  response  to  their  comments.  The  current  version  of  the  manual 
is  intended  for  anyone  in  the  maritime  industry  who  is  responsible  for  the  development  of 
assessments  that  will  rely  upon  practical  demonstration  and  comply  with  IMO  STCW  Code 
requirements. 

The  manual  provides  a  step-by-step  method  for  assessment  development.  It  is  organized  around 
the  five  PBA  assessment  development  steps  (see  Figure  1),  with  each  step  further  divided  into 
additional  sub-steps.  Detailed  instructions  for  completing  each  sub-step  are  provided.  The 
manual  also  includes  several  decision  aids  and  heuristics  for  assessment  developers.  Throughout 
the  manual,  an  extended  example  is  provided,  based  upon  the  lookout  duties  sample  assessment, 
showing  how  to  apply  the  steps  of  the  assessment  development  process.  In  addition,  the 
complete  sample  assessment  for  lookout  duties  is  provided  as  an  attachment  to  the  manual.  A 
copy  of  the  manual  is  provided  as  Chapter  3,  A  Manual  for  Assessment  Developers. 

Reviewers  consistently  acknowledged  the  technical  and  practical  value  of  the  manual  as  a 
reference  and  aid  in  assessment  development.  Many  reviewers  also  expressed  concern  about  the 
complexity  of  the  assessment  development  process.  This  concern  typically  has  focused  more  on 
issues  concerning  the  level  of  effort  required  to  develop  the  large  set  of  assessment  procedures 
specified  by  the  STCW  Code,  rather  than  the  technical  complexity  of  the  process  itself.  That  is, 
all  reviewers  have  noted  that,  after  investing  some  time  in  studying  the  manual,  they  understand 
the  process  for  developing  assessment  procedures  and  are  comfortable  applying  it.  Flowever, 
these  individuals  also  recognize  that  a  significant  level  of  effort  will  be  required  to  develop  such 
procedures  for  the  range  of  proficiencies  identified  in  the  STCW  Code. 

In  addition,  reviewers  frequently  expressed  concern  about  the  consistency  of  terminology  among 
IMO  doctrine,  USCG  NVICs,  the  ISD  literature,  and  the  manual.  IMO  terminology  has  been 
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used  as  much  as  possible.  One  result  of  this  approach  has  involved  the  use  of  the  term 
“assessor,”  which  is  used  in  IMO  documents,  rather  than  the  term  “designated  examiner,”  which 
is  the  term  used  in  the  USCG  NVIC  6-97  (USCG,  1997B). 

Sample  Assessment  Procedures 

A  key  activity  in  the  present  PBA  research  effort  involved  the  development  of  a  set  of  sample 
assessment  procedures.  Faculty  members  from  Massachusetts  Maritime  Academy  prepared  two 
assessments  for  navigation  watch  proficiencies  -Performance  of  Lookout  Duties  and 
Performance  of  Helmsman  Duties.  Faculty  members  from  California  Maritime  Academy 
prepared  two  assessments  for  engineering  watch  proficiencies  -Preparing  the  Main  Engine  for 
Operation  and  Locating  Common  Faults  and  Preventing  Damage  to  Generators. 

The  academy  participants  prepared  the  sample  assessments  using  the  draft  workshop  materials 
with  only  limited  review  and  guidance  from  the  project’s  assessment  development  experts.  As 
part  of  the  assessment  development  process,  the  participating  faculty  members  conducted  trial 
applications  of  draft  assessment  procedures  with  academy  cadets  playing  the  role  of  candidates. 
As  a  result  of  their  work  in  developing  the  sample  assessments,  the  faculty  participants  provided 
critical  comments  regarding  the  technical  adequacy  of  the  PBA  method,  specific  areas  where 
more  complete  guidance  was  required,  and  potential  standards  for  assessment  procedure  content 
and  format. 

During  the  course  of  developing  the  four  assessment  samples,  several  standard  approaches 
toward  assessment  procedure  content  and  format  evolved.  Specific  areas  where  standard 
approaches  were  developed  included  assessment  control  sheets  and  worksheets,  examiner 
instructions,  candidate  instructions,  and  assessment  documentation.  These  standard  approaches 
are  reflected  in  both  A  Manual  for  Assessment  Developers  and  the  sample  assessments 
procedures.  Appendices  A  through  D  provide  the  four  sample  assessments. 

In  developing  the  assessments,  a  number  of  issues  were  raised  that  have  yet  to  be  fully 
reconciled.  First,  there  is  the  issue  regarding  the  appropriate  level  of  detail  of  performance 
objectives,  measures,  and  standards.  The  current  set  of  samples  has  a  consistent  level  of  detail, 
in  which  all  specific  actions  required  of  a  candidate  are  defined.  An  alternative  approach  would 
be  to  provide  more  general  guidance  with  less  detailed  definitions  of  performance  measures  and 
standards.  In  this  case,  existing  operating  procedures  would  be  assumed  to  provide  the  basis  for 
defining  measures  and  standards.  The  present  authors  are  concerned  that  a  less  detailed  approach 
would  be  likely  to  reduce  assessment  validity  and  reliability.  Typically,  existing  operating 
procedures  do  not  provide  adequately  comprehensive  and  accurate  performance  steps  and 
standards  to  support  valid  and  reliable  assessment.  However,  a  reduced  level  of  detail  is  viewed 
by  some  as  necessary  in  order  to  deal  with  the  large  number  of  assessments  that  must  be 
developed.  Further  information  regarding  this  issue  will  be  obtained  during  the  trial  application 
of  the  sample  assessments  currently  underway  at  the  participating  commercial  shipping  company. 

A  second  issue  concerns  subject  matter  expert  review  of  assessment  procedures.  In  an  industry 
with  a  high  degree  of  individualism,  it  is  not  surprising  that  one  of  the  areas  where  assessment 
developers  confronted  difficulties  was  in  obtaining  consensus  agreement  among  other 
operational  experts  regarding  the  suitability  of  the  developed  procedures.  In  developing  the 
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sample  assessments,  the  participating  faculty  had  difficulty  in  convening  a  group  of  experts  and 
reaching  consensus  on  the  proficiency  measures,  standards,  and  scoring  procedures.  This  issue, 
as  well  as  those  identified  earlier  in  this  section,  is  reviewed  and  discussed  in  the  following 
section  of  this  report. 
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CONCLUSIONS  AND  RECOMMENDATIONS  _ 

The  major  objectives  of  this  phase  of  the  project  were  to  refine  the  PBA  method  and  to  prepare 
materials  to  assist  the  industry  in  applying  it.  The  approach  to  achieving  these  objectives 
involved  introducing  the  method  to  a  varied  sample  of  industry  representatives  through  the 
workshops  and  inviting  some  of  them  to  participate  in  trial  applications  of  the  method.  This 
approach  provided  the  opportunity  to  refine  both  the  general  method  and  the  materials.  The 
materials  provided  here  -  workshop,  manual,  and  sample  assessment  procedures  -  have  benefited 
from  multiple  reviews,  multiple  trial  applications,  and  substantial  comment. 

Conclusions 

The  PBA  method,  as  described  in  this  report,  is  compliant  with  the  requirements  of  the  STCW 
Code ,  USCG  NVICs,  ISD,  and  the  best  available  industry  practices.  The  industry  representatives 
who  participated  in  the  project  and  came  to  understand  the  method  were  quite  positive  about  the 
value  of  the  general  method  in  enhancing  the  rigor  of  assessment  procedures.  They  are  currently 
applying  it  in  their  own  situations  -  sometimes  with  their  own  adaptations.  If  the  PBA  method  is 
to  be  applied  more  broadly  in  the  industry,  these  materials  must  be  more  widely  available. 

As  the  trial  assessments  were  developed  and  circulated  for  review,  several  difficult  technical 
issues  were  identified  that  have  not  been  resolved.  One  issue  is  uncertainty  about  the  level  of 
detail  appropriate  in  the  development  of  an  assessment  procedure.  How  much  detail  is  necessary 
in  describing  the  assessment  objectives  and  criteria  to  assure  reliability  and  validity?  How  much 
should  be  left  to  the  judgment  of  the  assessor?  There  is  certainly  no  one  answer  to  the  question 
of  how  much  detail  is  needed.  Major  considerations  will  include  the  particular  proficiency  in 
question  and  the  circumstances  under  which  the  assessment  will  be  conducted.  The  trial 
applications  being  conducted  with  the  commercial  shipping  company  will  provide  some  insight 
into  this  issue  in  at  least  one  real  context.  Participants  from  the  shipping  company  will  provide 
suggestions  based  on  their  experience  with  the  assessments  regarding  the  level  of  detail 
appropriate  in  the  assessment  materials  and  the  level  of  detail  required  in  assessment  instructions, 
measures,  standards,  and  scoring  procedures. 

A  second  issue  is  the  difficulty  in  convening  expert  panels  and  obtaining  consensus  about  the 
appropriate  assessment  objectives,  measures,  and  standards.  Some  of  the  difficulty  in  holding 
and  conducting  such  meetings  stemmed  from  a  lack  of  familiarity  with  the  PBA  method  among 
those  who  were  asked  to  serve  as  experts.  It  is  expected  that  broader  implementation  of  the  PBA 
method  would  mitigate  much  of  this  problem.  However,  there  is  also  a  related  issue  of  experts 
representing  different  organizations  and  operational  settings.  Different  operational  settings  are 
often  associated  with  legitimate  differences  in  the  performance  requirements  of  mariners.  In 
these  cases,  the  differences  must  be  either  reconciled  through  changes  in  operations,  or 
recognized  and  reflected  in  different  assessment  objectives,  measures,  and  standards. 

An  additional  issue  concerns  the  level  of  effort  required  to  prepare  a  full  set  of  assessments  that 
will  be  responsive  to  the  full  range  of  proficiencies  identified  in  the  STCW  Code.  It  must  be 
recognized  that  the  broad  sets  of  proficiencies  defined  in  the  STCW  Code  resulted  from  an 
extensive  effort.  Providing  the  detailed  assessment  procedures  for  all  proficiencies  is  a  much 
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more  extensive  effort.  However,  several  factors  could  help  reduce  the  overall  level  of  effort  and 
provide  adequate  time  for  the  systematic  development  and  implementation  of  procedures.  First, 
many  of  the  operating  procedures  and  standards  upon  which  these  assessments  will  be  based 
already  exist.  Shipping  companies  and  individual  vessels  typically  have  standard  operating 
policies  and  procedures  for  their  crews,  which  can  serve  as  the  starting  point  in  developing 
assessments.  Second,  in  many  cases,  assessment  procedures  can  be  developed  and  shared  within 
and  between  organizations.  Schools  and  academies  could  share  assessment  procedures,  as  could 
similar  ships.  Differences  in  operating  requirements  that  dictate  differences  in  the  assessment 
objectives,  measures,  and  standards,  will  typically  represent  variations  across  similar  assessments 
and  much  of  the  content  and  format  of  assessments  could  still  be  shared. 

Early  in  the  project’s  examination  of  PBA,  it  became  apparent  that  developing  and  conducting 
assessments  were  separate  functions,  likely  to  be  performed  by  separate  individuals  and  even 
separate  organizations.  Further  project  efforts  will  examine  the  issues  of  conducting  mariner 
assessments,  including  planning  for  the  actual  conduct  of  a  procedure,  the  degree  of  detail 
appropriate  for  the  assessment  forms,  and  the  amount  of  familiarization  or  training  needed  by  the 
assessor.  A  separate  report  provides  materials  for  use  in  conducting  assessments. 

Recommendations 

For  the  most  effective  implementation  of  the  project’s  findings  and  products,  the  following 
USCG  and  industry  actions  are  recommended. 

•  The  USCG  should  encourage  the  maritime  industry  in  the  use  of  the  PBA  method,  as 
described  in  the  project  reports  and  materials.  As  a  first  step,  the  materials  in  this  report 
should  be  made  widely  available,  on  the  USCG  STCW  website 
(http://www.uscg.mil/stcw)  and  through  the  National  Technical  Information  Service 
(NTIS).  When  the  industry  becomes  more  familiar  with  the  PBA  method,  it  can  serve  as 
the  basis  for  a  new  NVIC.  In  addition,  the  USCG  should  submit  the  PBA  supporting 
materials  to  the  IMO  subcommittee  on  Standards  for  Training  and  Watchkeeping. 

•  The  USCG  should  encourage  further  familiarity  with  the  PBA  approach  by  USCG  staff, 
as  a  tool  for  discussions  of  assessment  with  the  industry  and  for  the  review  of  procedures 
submitted  for  approval. 

•  The  USCG  should  encourage  the  review  and  use  of  the  PBA  method  and  materials  by 
those  groups  who  are  dealing  with  the  important  technical  issues  of  assessment,  such  as 
the  Maritime  Academy  Simulator  Committee  (MASC),  the  Merchant  Marine  Personnel 
Advisory  Committee  (MERPAC),  and  academy  committees  appointed  by  the  Maritime 
Administration  (MARAD)  to  address  STCW  issues.  The  materials  can  provide  a 
common  approach  and  a  common  basis  for  discussion. 

•  Those  in  the  industry  who  are  responsible  for  training  and  assessment  of  mariner 
proficiency  in  academies,  training  schools,  and  shipping  companies  should  make  use  of 
the  PBA  method  and  the  materials  presented  here  as  a  guide  for  their  own  development  of 
assessment  procedures.  The  materials  are  especially  appropriate  for  inclusion  in  train- 
the-trainer  courses. 
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A  WORKSHOP  FOR  ASSESSMENT  DEVELOPERS 


This  chapter  contains  an  outline,  slides,  and  exercises  for  a  two-day 
workshop  on  developing  mariner  assessments. 


This  report  may  be  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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Developing  Mariner  Assessments: 
Workshop  Outline 


Subject  Area 


Hours 


Lectures  Exercise 


1 .0  Introduction  to  Assessment  Development 
Process 

1 . 1  Workshop  staff  and  participant  introductions 

1 .2  Review  of  research  project  context,  instructional  theory,  and 
recent  guidance  from  various  sources 

1 .3  Workshop  scope,  audience,  objectives,  and  agenda 

1 .4  Assessment  uses,  successes,  and  examples 

1 .5  Overview  of  assessment  development  process 

.25 

.25 

.25 

.15 

.10 

1.00 

2.0  Validity  and  Reliability 

2. 1  Definition  of  validity 

.25 

2.2  Overview  of  issues  relevant  to  validity 

.10 

2.3  Definition  of  reliability 

.25 

2.4  Overview  of  issues  relevant  to  reliability 

.10 

2.5  Discussion  of  validity  and  reliability  of  mariner  assessments, 

JO 

,50 

including  practical  experience  with  reliability  and  validity 

1.00 

.50 

(Exercise  1) 

3.0  Step  1 :  Specify  Assessment  Objectives 

3.1  Selection  of  proficiency  to  be  assessed 

.10 

3.2  Identification  of  regulatory  requirements 

.25 

3.3  Analysis  of  job  requirements 

.40 

3.4  Specification  of  assessment  objectives 

.25 

3.5  Examples  of  assessment  objectives 

.25 

3.6  Discussion  of  specifying  assessment  objectives,  including 

,25 

,50 

practical  experience  with  specifying  assessment  objectives 

1.50 

.50 

(Exercise  2) 

4.0  Step  2:  Determine  Assessment  Methods 

4. 1  Overview  of  basic  assessment  methods 

.25 

4.2  Pros  and  cons  of  basic  assessment  methods 

.25 

4.3  Overview  of  steps  and  issues  in  determining  assessment 

.25 

methods 

4.4  Examples  of  assessment  methods 

.25 

4.5  Discussion  of  determining  assessment  methods,  including 

25 

25 

practical  experience  with  determining  assessment  methods 

1.25 

.25 

(Exercise  3) 
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5.0  Step  3:  Specify  Assessment  Conditions 

5.1  Overview  of  steps  involved  and  issues  to  consider 

5.2  Conditions  requiring  specification 

5.3  Examples  of  assessment  conditions 

5.4  Discussion  of  specifying  assessment  conditions,  including 
practical  experience  with  specifying  assessment  conditions 
(Exercise  4) 

.25 

.10 

.25 

25 

0.85 

A0 

.40 

6.0  Step  4:  Develop  Proficiency  Criteria 

6. 1  Part  1 :  Developing  performance  measures 

6.2  Overview  of  measurement  definition  and  procedures 

.25 

6.3  Definition  of  performance  measures 

.25 

6.4  Types  of  measurement 

.25 

6.5  Examples  of  measurement  types 

.25 

6.6  Discussion  of  developing  performance  measures,  including 

25 

J5 

practical  experience  with  developing  performance  measures 

1.25 

.75 

(Exercise  5) 

6.7  Part  2:  Developing  performance  standards 

6.8  Definition  of  performance  standards 

.25 

6.9  Example  of  performance  standards 

.25 

6.10  Definition  of  scoring  procedures 

.25 

6. 1 1  Examples  of  types  of  scoring  procedures  and  standards 

.25 

6. 1 2  Discussion  of  developing  performance  standards,  including 

25 

25 

practical  experience  with  developing  performance  standards 

1.25 

.75 

(Exercise  6) 

7.0  Step  5:  Prepare  Assessment  Materials 

7. 1  Rationale  and  process  for  preparing  materials 

.25 

7.2  Example  assessment  form 

.25 

7.3  Documentation  to  be  prepared 

.25 

7.4  Example  materials 

.25 

1.00 

8.0  Summary 

8.1  Review  of  steps  involved  in  developing  mariner  assessments 

.25 

8.2  Summary  and  discussion  of  goals  for  the  workshop 

25 

0.50 

TOTAL 

9.60 

3.15 
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Developing  Mariner  Assessments: 

A  Practical  Workshop  for 
Assessment  Developers 


Workshop  Staff  Introductions 


Participant  Introductions 


Please  tell  us  briefly: 

•  Your  name 

•  How  you  are  related  to  the  maritime 
industry 

•  What  you  most  want  to  gain  from  the  course 


Project  Context 


•  USCG  and  industry,  increased  concern  for 
mariner  proficiency  in  performing 
shipboard  responsibilities 

•  Project  objective  to  assist  USCG  and 
industry  by  providing  tool  for  assessment 

•  Considered  instructional  theory,  recent 
USCG  guidance,  1995  I  MO  STCW 
amendments 


Instructional  Theory 


•  For  training,  performance-based  objectives: 
conditions,  behaviors,  standards  (as 
example:  I  MO  model  courses) 

•  For  assessment,  recent  practice: 

-  expert  judgment  of  skills,  valid 

-  paper  test  of  knowledge,  reliable 

•  Develop  performance-based  assessment 
procedures,  valid  and  reliable 


Recent  USCG  Guidance 


•  On  company  roles  and  responsibilities 
regarding  STCW  (NV1C  4-97) 

•  On  Qualified  Instructors  and  Designated 
Examiners  (NVIC  6-97) 

•  On  USCG/industry  partnerships  through  a 
Quality  Standards  System  for  workshops 
and  training  programs  (NVIC  7-97) 
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1995  IMO  STCW  Amendment 

IMO  MSC  Circular,  May  1998 

•  Requires  assessment  of  mariner  competence 

•  Formal  assessment  of  competence  under 

by  demonstration  of  performance 

authority  of  approved  center 

•  Provides  required  proficiencies, 

•  Conduct  of  assessment  of  proficiencies  by 

understanding,  skills  by  mariner  role 

qualified  assessor  onboard  ship 

•  Provides  guidance  on  assessment  methods: 

•  Guidance  for  developing  and  conducting 

shipboard,  simulator,  laboratory 
•  Provides  guidance  on  assessor,  on  using 
established  measures  and  standards 

assessments  (similar  to  project  approach) 

Drvrlopinj  M»rinff  AMTMmcntJ  7  of  1  3" 

Workshop  Scope  and  Audience 

Workshop  Objectives 

•  Scope:  Tool  for  developing  procedures  for 

•  Understand  validity  and  reliability 

assessment  of  mariner  proficiencies 

•  Demonstrate  development  of  performance- 

•  Audience 

based  assessment  objectives 

-  USCG  reviewers 

-  USCG-accepted  third-party  reviewers 

•  Demonstrate  specification  of  methods, 

-  Training  program  assessment  developers 

conditions,  and  exercises 

-  Qualified  instructors  and  designated  examiners 

•  Understand  performance  measurement 

NOTE:  This  workshop  does  not  now  meet  any  IMO.  STCW,  or 

•  Demonstrate  development  of  standards 

USCG  requirements. 

*  Demonstrate  preparation  of  materials 

Dnrrk>ptn|  M*rinrr  Ai wiwirtiu  9  of  1 5' > 

Developing  Procedures  for 

Workshop  Agenda 

Day  1 

Assessment  of  Mariner  Proficiency 

-  The  Assessment  Development 

-  Assessment  Validity  and 

Reliability 

Step  1 :  Identify  Assessment  JL 

objec,ives 

Step  2:  Determine 

Assessment  Methods 

rwi*pin,  Am,,.*,™..  nr-n?" 
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Workshop  Agenda 


Instructional  Development  and 
Assessment  Development 


Step  3:  Specify  Assessment  Conditions 
Step  4:  Develop  Performance  Measures  and 
Standards 

Step  5:  Prepare  Assessment  Materials 
Summary/Overview 


Develop 

Aswssment 

Procedures 


Assess 

Performance 


Common  Uses  of  Assessment 


Assessment  Success  Stories 


Equipment  purchases 
Judging  animals  in 
competitions 
Comparing  athletic 
performances 

Evaluating  the  performance 
of  employees 


i* 


Telecommunications 
industry 
Pilot  training 
Fire  fighting 
Tank  crew  training 


Example  of  Employee  Assessment 


Hamburger  Heaven  Food  Server 

Assessment  Objective 

Performance  Standard 

Friendly  and  courteous 

Smiles  and  greets  every  customer 

Speedy  service 

More  than  40  orders  per  hour 

Accurate  charging 

Less  than  1%  errors 

Accurate  order  filling 

Less  than  10%  errors 

Developing  Manner 

Five  Steps  of  Assessment 
Development  _ 


[Stepl:  Identify  assessment  objectives] 

Step  2:  Determine  assessment  methods] 

Step  3:  Specify  assessment  conditions'] 

Step  4:  Develop  proficiency  criteria  | 

^fi|  Step  5:  Prepare  assessment  materials] 
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Validity  and  Reliability  of 
Mariner  Assessments 


Validity 


•  Validity  refers  to  the  idea  of  assessing  the 
performance  of  interest 

•  Example: 

-  Head  diameter  as  an  assessment  of  intelligence 
has  low  validity 

-  Head  diameter  as  an  assessment  of  a  mariner  s 
ability  to  get  his/her  head  in  and  out  of  a  port 
hole  has  high  validity 


Valid  Assessments 


Case  Study:  Rules  of  the  Road 


Include  the  work  equipment  (controls, 
displays,  tools)  that  are  required  for 
successful  performance 
Provide  the  cues  from  the  job  environment 
that  are  used  on  the  job  to  determine 
appropriate  actions 

Assess  skills  and  knowledge  that  are  critical 
to  successful  performance 


Case  Study:  Rules  of  the  Road 


Reliability 


Assessment  Method  Validity  Issues  _ _ 


Multiple-choice  test  Positive:  Covers  broad  range  of  topics 
,tems  Negative:  Test  items  can  be  memorized.  Assesses 

recall  of  information,  not  application 

Situation  Assessment  Positive:  More  directly  resembles  bridge 
watchkeeping  requirements 
Negative:  Does  not  assess  collision  avoidance 

Collision  Avoidance  Positive:  Resembles  additional  bridge  watchkeeping 
requirements 

Negative:  Does  not  assess  application  of  Rules  on  a 


Reliability  refers  to  the 
consistency  of 
assessment 
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Reliable  Assessments 


•  Consist  of  procedures  that  are  easily 
repeated  between  assessments  and  assessors 

•  Have  well-defined  performance  measures 

•  Have  well-defined  procedures  for 
comparing  performance  to  standards 


Developing  Mariner  a  of  (in 


Case  Study: 

Rules  of  the  Road 

Assessment  Method 

Reliability  Issues 

Multiple-choice  Items 

Positive:  Items  are  repeated,  minimal  assessor 
involvement,  well-defined  scoring 

Negative:  May  be  misinterpreted 

Situation  Assessment 

Positive:  Scenario  is  repeated,  minimal  assessor 
involvement,  well-defined  scoring 

Negative:  Limited  number  of  scenarios,  artificial 
setting  may  make  interpretation  difficult 

Collision  Avoidance 

Positive:  Scenarios  always  start  the  same 

Negative:  Scenarios  vary  with  maneuvers,  well- 
defined  scoring  is  difficult  (but  feasible) 

Drvrlopmt  Mariner  AtsetimctrU 

*nf1w 

Validity  and  Reliability  of  Mariner 
Assessment 

Exercise  1:  Understanding 

Assessment  Validity  and  Reliability 

•  Are  valid  and  reliable 

•  Hamburger  Heaven  Server 

assessment  procedures  a 

Training  and  Assessment 

practical  goal  in  the  maritime 

-  Candidates  receive  a  full  day  of 

1  industry? 

training  at  Hamburger  Limbo. 

1  •  What  are  the  main  obstacles 

-  Candidates  must  successfully 

H  to  valid  and  reliable 

complete  Hamburger  Limbo 
assessment  in  order  to  advance  to  a 

assessment  of  mariners? 

probationary  position  at 

^  •  How  can  we  best  address 

Hamburger  Heaven. 

these  obstacles? 

ftnrlopmf  Manner  Aurcmcnt)  27  C*f  1  JR 
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Exercise  1 ,  continued 

Exercise  1 ,  continued 

The  first  four  hours  at  Hamburger  Limbo  is  a 

The  four  standards  of  Hamburger  Heaven 

series  of  lectures  addressing: 

server  performance  are: 

-  Hamburger  Heaven  mission  and  vision 

-  Smile  and  greet  every  customer 

-  Hamburger  Heaven  profit  goals 

-  40+  orders  taken  per  hour 

-  Worker  safety 

-  Less  than  1%  charging  errors 

-  Server’s  attire  and  hygiene  requirements 

-  Less  than  1 0%  order  filling  errors 

-  Four  standards  of  Hamburger  Heaven  server 
performance 

rxvtkipinf  Marine*  Antaounti  SuflJii 

Developing  M  inner  Aiminnli  lilofljd 
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Exercise  1,  continued 

Exercise  1 ,  continued 

The  final  four  hours  in 
Hamburger  Limbo  is  a 
four-part  assessment: 

1 .  Computer 

y9 

•  We  will  assist  in  forming  small  work  groups 

•  Each  group  should  review  the  four 
assessment  procedures  and  provide 
comments  and  recommendations  on: 

-  Positive  &  negative  validity  issues 

-  Positive  &  negative  reliability  issues 

-  Recommended  assessment  procedure 
improvements  for  validity  and  reliability 

administered 
multiple-choice  test 

2.  Greeting  assessment 

3.  Order  taking 
assessment 

4.  Order  filling 
assessment 

DrvrU>|>m|  Manner  AiKummti  31  of  1 50 
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Step  1 :  Specify  Assessment 
Objectives 


DrvrViJhnf  MiHnrr  Awvmo»l«  Mr>ri)l> 


Steps  in  Specifying  Assessment 
Objectives 

Identify  Regulatory  Requirements 

1.  Select  proficiency  to  be  assessed 

•  Identify  specific  requirement  in  STCW  code 

•  5  levels  of  requirements  in  STCW  code: 

2.  Identify  regulatory  requirements 

-  Chapter 

3.  Analyze  job  requirements 

-  Table 

4.  Describe  individual  assessment  objectives 

|)rvrlopm|  Mini***  33  of 

-  Function 

-  Competence 

-  Proficiency 

Drvrlcprnf  Maram  AiKiimrwia  36  of  1  S,( 
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STCW  Code 


Competencies:  Function 
onboard  ship 

Knowledge,  Understanding, 
Proficiency:  Specific  skill 
and  knowledge  requirements 


$7  of  J  5ft 


Analyze  Job  Requirements 


•  Identify  job  requirements  using  available 
sources 

•  Document  sources 

•  Identify  critical  job  requirements 

•  Make  sure  the  assessment  procedure  meets 
STCW  requirements  and  is  practically 
feasible 

Drvtlopmg  Munnrr  A  mil  .WnflJn 


Sources  of  Mariner  Task  Information 

Job  or  Task  Analysis 

•  STCW  competencies,  knowledge, 
understanding,  and  proficiency 

•  U.S.  Code  of  Federal  Regulations 

•  Job  or  task  analysis 

•  Subject  matter  expert  knowledge  and 
judgment 

•  Technical  manuals 

•  Bargaining  agreements 

Developing  Marion  Aiiewnrtft*  59  of  15ft 

Job  Description:  General  responsibilities  and 
duties 

Task  Description:  Specific  activities  with 
identifiable  beginning,  action,  output 

Task  Criticality:  Identification  of  tasks 
critical  to  successful  job  performance 

-  Frequency 

-  Importance 

Drvtloping  M  viiwi  AsiMimmB  nf  I5n 

Collection  of  Job/Task  Data 

Subject  Matter  Experts  (SMEs) 

1.  Determine  the  job  to  be  analyzed  - - - 

2.  Prepare  the  data  collection  plan 

3.  Gather  initial  data  \  f  ‘  ] 

4.  Prepare  data  collection  forms  Hk 

5.  Select  sample  and  arrange  data 
collection 

6.  Collect  data 

7.  Consolidate  data  into  lists  of  tasks  with 
criticality 

8.  Validate  task  list  and  determine  skill 
and  knowledge  requirements 

9.  Finalize  list 

•  Provide  initial  input  in  task  analysis 

•  Provide  a  substitute  or  shortcut  to  formal 
task  analysis 

•  SME  sources 

-  Current  jobholders 

-  Supervisors 

-  Instructors 

-  Former  jobholders 

-  Others  familiar  with  job 

Developing  Mariner  A«snimcni«  ofl 
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Describe  Individual  Assessment 
Obi  ectives _ _ 


Assessment  Objectives  . . . 


Assessment  Objectives  for 
LookoutProcedure^continued^^_ 

6.  Demonstrate  lookout  techniques  &  make  lookout 
reports  in  restricted  visibility  during  daylight  or  at 
night 

7.  Demonstrate  the  use  of  lookout  equipment 

8.  Demonstrate  man  overboard  procedures 

9.  Demonstrate  lookout  watch  relief  procedures 


Comparison  of  Simulator  and 
Shipboard  Assessment 


How  would  you  need  to 
identify  assessment  objectives 
differently  if  you  were 
assessing  via  simulation  and 
shipboard  assessment? 

-  ARP  A  example 

-  Lookout  Procedures  example 


Developing  Mariner  Aneumrrrfi 
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Exercise  2:  Specifying  Assessment 
Objectives^ ^ ^ ^ — — m 

•  Goal:  To  specify  six  lifeboat  launching 
assessment  objectives 

•  Tools: 

-  STCW  and  CFR  Lifeboat  Competencies 

-  Lifeboat  Launching  Procedure 

-  Examples: 

•  Lookout  Procedures 

•  ARPA 

-  Assessment  Objectives  Worksheet 

Drvdopiag  Mariner  AMttftttnli  H 


Exercise  2,  continued 

•  To  get  started: 

-  Review  the  STCW/CFR 

0^, 

competencies 

-  Review  the  lifeboat  task  data 

-  Use  the  above  &  your  knowledge 
of  best  practices  as  the  basis  for 
your  objectives 

-  Use  the  Lookout  and  ARPA 
examples  as  a  guide  for  the  format 
of  your  objectives 

Dcvtlnping  Mariner  AftnamraU  JIrflW 

Exercise  2,  continued 


♦  Your  assessment  objectives  should  include: 

-  Action 

-  Preliminary  conditions 

-  Preliminary  standard(s) 

•  Example: 

-  Given  relatively  complex  and  varied  traffic 
{condition),  using  ARPA  calculate  the  required 
new  course  for  own  ship  {action)  to  maintain  a 
minimum  CPA  between  own  ship  and  other 
vessels  {standard). 

Dr*Hflpin|  Mariner  Arrrttmtnri 


Exercise  2,  continued 

•  Use  the  overhead 
transparency  worksheet  to 
record  your  group’s 

answers. 

•  Select  a  spokesperson  to 

•  1 

1  tell  the  larger  group  your 

assessment  objectives. 

\  •  You  have  30  minutes  for 

this  exercise 

Developing  M»rinrr  Aftewmenb 
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Exercise  2:  Example  Solution 

Step  2:  Determine 

•  Many  correct  responses 

•  Solution  gives  example  of  6  possible 
objectives 

•  Example  objective: 

-  Given  a  stowed  lifeboat  and  4  cadets 
(conditions),  person  in  charge  uses  proper 
commands  and  terminology  (standard)  to  carry 
out  inspections  of  lifeboat,  falls  and  davit 
(action). 

l)»vrlop.n|Mw*m  A»»r«l»tm.  J3eflJ» 

Assessment  Methods 

Steps  in  Determining  Assessment 
Methods 

Basic  Assessment  Methods 

1.  Identify  alternative  assessment  methods 

•  Written  or  oral  test  questions 

•  Simulation  using  mock  equipment 

2.  Review  advantages,  disadvantages,  and 

•  Simulation  using  actual  equipment 

feasibility  of  alternatives 

•  Shipboard  assessments 

3.  Consider  performance  to  be  assessed  and 
review  validity  and  reliability  issues 

4.  Determine  assessment  method 

IM-rio[Hn,  M  Anr«mrr,li  JlofU" 
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Written  or  Oral  Test  Questions 

Characteristics  of  Simulation 

•  Environment 

•  Controls 

0^  •  Open-ended 

•  Displays 

•  Fill-in  (he  blank 

•  Processing  Characteristics 

DrvtV*ptD|  Manurr 

-  Hydrodynamics 

-  Mimicry  of  electronic  processing 
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Shipboard  Assessment 


Pros  and  Cons  of  Assessment  Methods 


•  Actual  or  dedicated  training/assessment 
equipment 

•  Operational  or  non-operational  conditions 

•  More/less  demanding  operational  conditions 


Drvrbp.ni  Mirir.fr  Aucttmtsi 


Consider  Performance  to  be 
Assessed ^ — 

•  Review  assessment  objectives 

•  Consider  task  initiation 

•  Consider  the  performance  product 

•  Consider  the  performance  process 
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Alignment 

Method 

Pro* 

Con* 

Written  or  oral 
test  questions 

•  Easy  to  create 

•  Rasy  to  standardize 

•  Rasy  to  test  many  candidates 

•  Good  for  measuring  knowledge 

•  May  have  poor  similarity  to  work 
environment 

•  Candidate  needs  good  verbal  skills 

Simulation 
using  mock 
equipment 

Simulation 
using  actual 
equipment 

•  Highconirol 

*  Safe 

•  High  control 

♦  Safe 

*  May  have  good  work  similarity 

•  May  be  expensive 

•  May  have  poor  work  similarity 

»  Substantial  assessor  requirements 

•  May  be  expensive 

•  Substantial  assessor  requirements 

Shipboard 

•  May  be  inexpensive 

•  Uses  actual  work  setting 

•  Substantial  assessor  requirements 

•  Operational  scheduling  eonfliets 

•  DilTicult  to  standardize  and  score 

•  Task  may  be  infrequent  or  dangerous 

Drvrkrpii>|M«racT  Aimmowiu* 


Review  Validity  and  Reliability 
Issue^^ — 

•  Review  realism  (validity)  requirements 

-  Environment/setting 

-  Controls 

-  Displays 

-  Processing  Characteristics 

•  Review  control  (reliability)  required 

-  Key  skill  and  knowledge  requirements 

-  Variability  of  task  conditions 

-  Comparability  of  assessment  conditions 

Developing  MirintrAummfnu  W  of  I  JO 


Review  Feasibility  of  Alternatives 


•  Can  performance  be  measured  via  written 
assessment? 

•  Can  performance  be  easily  and  safely 
elicited  onboard? 

•  Can  a  simulator  replicate  conditions  closely 
enough  to  cue  the  behavior? 

•  Which  is  easier,  cheaper,  safer,  more  legally 
defensible? 

Developing  Marc  it  65  of  W 


Assessment  Method  Example: 

Rules  of  the  Road 

Rules 

Prepared  Written 

Knowledge 

Questions 

Situation 

Mock  Equipment  —Highly 

Assessment 

Controlled  View  of 

Simulated  Traffic 

Collision 

Avoidance 

Mock  Equipment  -  Control  of 
Vessel  in  Moderately 

Controlled  Simulated  Traffic 

Developing  Miriner  Assmmcn*  n 
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Assessment  Method  Example: 

ARPA  Operation 

Assessment  Method  Example: 

Lookout  Procedures 

Knowledge  of  Prepared  Written 

Processing  Questions 

Capabilities 

Set-up,  Situation  Actual  Equipment  in  Lab  — 

Assessment,  Highly  Controlled  View  of 

Parallel  Indexing,  ^  Simulated  Traffic 

Warnings  and  Tests 

Actual  Equipment  in  Lab  — 

Collision  Avoidance  Control  of  Vessel  in  Moderately 

Controlled  Simulated  Traffic 

Developing  Marino  Ai«eijmerm  67  of  1 5t> 

•  Assessment  Requirements 

-  Knowledge  of  lookout  duties  &  procedures 

-  Correct  use  and  operation  of  lookout  techniques, 
reporting,  &  equipment 

•  Assessment  Constraints 

-  Must  not  interfere  with  vessel  safety 

-  Equipment  must  be  present  &  operational 

-  Conditions  of  clear  (day,  night)  &  restricted  visibility 

•  Assessment  Method  Selection 

-  Written  test  to  assess  knowledge  of  duties 

-  Shipboard  or  simulator  assessment  of  operations 

[>vrkipinj  Marin At ifiimrfttj  W  i*f  1 3** 

Exercise  3:  Determining 

Assessment  Methods 

Exercise  3,  continued 

•  Goal:  To  determine  the  appropriate 
assessment  methods  for  six  lifeboat 
launching  assessment  objectives 

•  Tools: 

-  “Pros  and  Cons  of  Basic  Assessment  Methods” 

-  Lookout  Procedures  example 

-  Assessment  methods  worksheet 

•  To  get  started: 

-  Remember  the  basic  assessment  methods: 

•  Written  or  oral  tests 

•  Simulation  in  controlled  lab  using  mock  equipment 
(e.g.,  PC-based  interface) 

•  Simulation  in  controlled  lab  using  actual  equipment 

•  Shipboard  assessment 

-  Review  the  pros  and  cons  of  each  method 
(Attachment  1) 

DwafepingMarncrAitrunwrta  6»  nfl  JO 
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Exercise  3,  continued 

Exercise  3  -  Summary 

*  Considerations  in  assessment  method 

•  Record  your  group’s 

selection: 

answers  on  your  overhead  Ml 

-  Appropriateness  of  written,  knowledge-based 

transparency  worksheet  f  \ 

assessment 

•  Select  a  spokesperson  to  \  /  jfk 

report  your  assessment  J 

methods  1  ^ 

-  Availability  of  actual  equipment  (lifeboat,  davit) 

-  Operational  safety 

-  Possibility  of  damaging  vessel’s  lifesaving 

•  You  have  15  minutes  ^  { 

equipment 

-  Ability  to  vary  conditions  on  equipment 

Drvfk*pir|  Mtrinrr  of  151' 
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Step  3:  Specify  Assessment 

Steps  in  Specifying  Assessment 
Conditions 

Conditions 

1.  Describe  operational  settings  and  scenarios 

2.  Specify  oral  and  written  test  questions 

3.  Provide  the  cues  used  in  selecting  and 
executing  performance  (validity) 

4 .  Standardize  assessment  conditions 

(reliability) 

Developing  Mnrintr  AitMimmB  M  ”r ' 1,1 

Issues  to  Consider 


•  What  must  be  presented  to  the  candidate  as 
cues  to  initiate  task? 

•  What  must  be  presented  to  the  candidate  to 
make  decisions? 

•  What  actions  must  be  taken  by  the 
candidate? 

•  What  assessment  objectives  will  require 
written  or  oral  questions? 


Conditions  Requiring  Specification 


•  Candidate  orientation 

•  Equipment,  apparatus, 
and  tools 

•  Initial  equipment 
settings  or  scenarios 

•  Written  or  oral 
questions 


Assessment  Conditions  Example: 

Rules  of  the  Road 

Assessment  Conditions  Example: 

ARPA  Operation  . 

•  Candidate  orientation 

•  Candidate  orientation 

-  ARPA  set-up  instructions 

-  30-minute  orientation  of  workstation  operation 

•  Equipment,  apparatus,  and  tools 

-  Practice  session  with  workstation 

-  Full-scale  ARPA  simulator 

•  Equipment,  apparatus,  and  tools 

-  Vessel  cards 

-  PC-based  bridge  simulator 

♦  Initial  equipment  settings  or  scenarios 

•  Initial  equipment  settings  or  scenarios 

-  Situation  observation  scenarios  (first  parts) 

-  Situation  observation  scenarios  (first  parts) 

-  Collision  avoidance  (second  parts) 

-  Collision  avoidance  (second  parts) 

♦  Written  or  oral  questions 

•  Written  or  oral  questions 

-  Target  information  forms 

-  Factual  questions 

-  Factual  questions 

Devrloprat  M*ri*KT  77off5n 
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Assessment  Conditions  Example: 

Lookout  -  Lookout  Assessment  Objective  4 


Assessment  Objective 

4.  Demonstrate  lookout  techniques  and  make  lookout 
reports  in  dear  visihilit y  during  daylight 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator 
assessment 

Candidate  Orientation 

Assessor  briefs  candidate  on  assessment  methods, 
conditions,  and  standards 

Required  Equipment, 
Apparatus,  &/or  Tools 

Lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  binoculars, 
and  bearing  repeater  fitted  w /  bearing/  azimuth  circle, 
alidade,  or  pelorus.  The  lookout  station  should  be  clear, 
and  the  examiner  must  be  able  to  observe  activities. 

Initial  Conditions 

Conduct  assessment  in  clear  visibility  during  daylight. 
Ensure  reportable  objects  are  in  sight. 

Drtetnf»n|  M*iner  A*aet»rnnjt* 
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Comparison  of  Simulator  and 
^hi^boardAssessmen^^^^^^^ 

*  What  are  some  of  the  challenges  in 
controlling  conditions  for  simulator  and 
shipboard  assessment? 

-  Providing  the  cues  used  in  selecting  and 
executing  performance  (validity) 

-  Standardizing  assessment  conditions 
(reliability) 
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Exercise  4:  Specifying  Assessment 
Conditions 

Exercise  4,  continued 

•  Goal:  To  specify  conditions  for  your 

•  To  get  started: 

lifeboat  launching  assessment 

-  Review  the  lifeboat  launching  assessment 
objective  given  to  your  group 

•  Tools: 

-  Consider  the  preliminary  conditions  and 

-  Group  responses  to  Exercises  2-3 

method(s)  you  specified  for  this  objective 

-  Lookout  and  ARP  A  examples 

-  Use  the  Lookout  &  ARPA  examples  as  a  guide 

-  Assessment  conditions  worksheet 

•  ARPA  example  has: 

-  Detailed  exercise  conditions 

-  Assessor  and  candidate  instructions 

*'  °n5" 

Exercise  4,  continued 

Exercise  4,  continued 

•  ARPA  Example: 

-  Assumes  assessment  in  controlled  lab  using 
full-scale  functional  ARPA 

-  “Exercise  E”  is  one  of  7  exercises  candidate 

•  Candidate  assessed  on  objective  1 .6  - 
“Understanding  of  when  to  use  ground-  or  sea- 
stabilized  modes,  and  when  to  use  north-  up, 
course-up  and  head-up  presentations” 

•  Candidate  given 

performs 

-  full-scale  ARPA  simulator 

•  Scenario  is  programmed  ahead  of  time  into 

-  vessel  card 

simulator  by  assessor 

-  chartlet 

•  Narrow  channel  navigation  with  cross-current 

•  Own  ship  is  outbound 

•  One  threat  vessel  is  inbound 

-  exercise  instructions 

-  worksheet  to  record  vessel  data  &  response  to  written 
question 

DrvrVpTftg  Atmimrati  !U  nfl  Jll 
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Exercise  4,  continued 

Exercise  4,  continued 

•  Assessor  given: 

-  Instructions  for  how  to  program  simulator 
—  Instructions  for  how  to  administer  assessment 

•  Record  your  conditions  on  the  transparency: 

-  Identify  setting 

-  Identify  what  must  be  presented  to  candidate 
(display) 

—  Identify  candidate  action  items 

-  Identify  objectives  requiring  questions 

•  Select  a  spokesperson 

•  You  have  20  minutes 

0rvrk>pm|  *  ofljft 

Exercise  4:  Example  Solution 

Exercise  4:  Example  Solution, 

•  Assessment  to  be  conducted  at  sea  during 

•  Assessor  makes  sure  candidate  and  crew 

weekly  abandon  ship  drill 

members  have  lifejackets  on 

•  Weather  &  sea  conditions  conducive  to  safe 

•  Assessor  briefs  candidate  on  assessment 

assessment 

objectives  and  standards 

•  Assessment  begins  with  lifeboat  in  stowed 

•  Assessor  has  a  checklist  of  candidate  action 

position 

items  and  standards  of  performance 

•  Assessor  varies  equipment: 

•  Assessor  completes  checklist  as  candidate 

-  Unfastens  sea  painter  from  thwart 

performs  lifeboat  launching 

-  Secures  one  tricing  pendant  incorrectly 

lx  -il— jn  of  150 
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Step  4:  Develop  Proficiency 
Criteria 

Parti:  Develop  Performance 
Measures 


Measures  and  Standards 

Measure 

Standard 

Correct  charging 

99% 

Correct  order  filled 

90% 

CPA 

More  than  0.25  nm 

ARP  A  set-up  time 

Less  than  5  minutes 

Dtvrlopint  Mariner  Aimumerm 
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Developing  Performance  Measures 
&  Standards  tm 

Definition  of  Measurement 

•  Match  performance  |— 

•  Measurement  is  applying  rules  to  assign 
categories  or  numbers  to  different  levels  or 
degrees 

measures  to  the  r  . ^ 

assessment  objectives  - — - -  ] 

•  Refine  each  performance  m  Mjm-~ 

•  Categories  and  numbers  represent  levels 

•  Rules  help  make  categories  or  numbers 

measure  >  'fl 

At  * 

comparable 

Developing  Manner  9|  oflJO 
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Performance  Measures 


•  Performance  measures  are  procedures  for 
observing  and  recording  a  candidate’s 
performance 

•  Measures  can  be  objective  or  subjective 

•  Measures  can  focus  on  the  process  or 
product  of  performing  the  task 

•  Try  to  measure  the  performance  required  to 
meet  an  objective  in  more  than  one  way  to 
increase  validity  and  reliability 

t)«v«lopieg  Winner  ”  °f  '  Sf' _  _ 


Measurement  Types 

Type  of 
Measurement 

Activity 

Measure 

Objective 

Speed  Skating 

Time  to  Finish 

Subjective 

Rating 

Figure  Skating 

Technical  & 

Artistic  Merit 

Subjective 

Checklist 

Fast  Track 
Racing 

Finish  Race  or 

Not 

Drvt lopiftj  Mariner  Aii«»fiKT»ir 
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Types  of  Measurement 


Different  types  of  measurement  give 
you  different  amounts  of  information 

Types/Info. 

Amount 

Relative 

Group 

Different 

Performance 

Member 

Objective 

• 

• 

• 

Rating 

• 

• 

Checklist 

• 

«  »r  lit' 

Measuring  What 
You  Want  to  Measure 


Collision 


Rules  Knowledge  and  Application  Avoidance 


Situation 


H  Bridge  Watchkeeping 
BE  Rules  Assessment 
■  Rules  Knowledge  and  Application 
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Measuring  What 
You  Want  to  Measure 


9]  Rules  Assessment  Relevance 

m  Rules  Knowledge  and  Application 
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Example  of  Rating  Measure 


USMMA  Cadet  Watch  Team  Measure  Examples 


Measure 

Rating 

Compliance  with  Master’s/Standing  Orders 

o, 

1,2 

Proper  Preparation  for  Arrival 

0, 

1,2 

Proper  Internal  Communications 

0, 

1,2 

Proper  vhf  Procedures 

o, 

1,2 

Master/Engine  Room  Kept  Informed 

o, 

1,2 

Dfvttepim  Mwiatr  Aiicnmniti 


Comparison  of  Simulator  and 
Shipboard  Assessment _ 

•  How  would  you  need  to  develop 
performance  measures  differently  if  you 
were  developing  procedures  for  simulator 
and  shipboard  assessment? 


Example  of  Checklist  Measure 


Lookout  Assessment  Objective  4:  Demonstrate  lookout 
techniques  and  make  lookout  reports  in  clear  visibility 
during  daylight 

4.1  List  the  sightings  that  should  be  reported  when 
detected 

4.2  Report  sighted  objects  using  ship’s  bell 

4.3  Report  sighted  objects  verbally  using  degrees 

4.4  Report  sighted  objects  verbally  using  points 

4.5  Report  sky  objects  verbally  using  points 

4.6  Report  audible  target  verbally  using  points 

DWtlopinJ  M«rintr  AmthdictiH  9,<lfl,<> 


Example  of  Objective  Measure 


ARPA  —  Situation  Assessment 


Target  Data 

Measure 

Bearing 

Degrees  from  Actual 

Range 

Nm  from  Actual 

Course 

Degrees  from  Actual 

Speed 

Nm  from  Actual 

CPA 

Nm  from  Actual 

TCPA 

Seconds  from  Actual 

Exercise  5:  Developing 
Perfonnanc^ 

•  Goal:  To  develop  performance  measures 
for  up  to  six  lifeboat  launching  assessment 
objectives 

•  Tools: 

-  Your  group  responses  to  Exercises  2-4 

-  Rules  of  the  Road  example 

-  Lookout  duties  example 

-  Worksheet 
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Exercise  5,  continued 

Exercise  5,  continued 

•  To  get  started: 

-  Consider  your  lifeboat  launching  assessment 
objectives 

-  Define  specific  performance  measures  for  each 
objective 

M  •  See  Attachments  1  &  2  for 

sEi  examples  of  performance 

measures  for  other  types  of 
7^rTY  assessments 

-  Remember  that  measures  describe  observable 

V  J  •  Record  your  group’s  answers 

actions  or  outcomes  and  that  they  may  be 

]/  on  the  overhead  transparency 

objective  or  subjective 

f  \  •  Select  a  spokesperson 

_  Include  both  process  and  product  measures 

•  You  have  45  minutes 

-  Please  do  not  develop  standards  at  this  time 

[>tvtlopm|  Manner  |03  of  15* 
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Exercise  5: 

Example  Performance  Measures^  _ 

Step  4:  Develop  Proficiency 

•  Following  is  based  on  task  information  given  in  “Launching 

Criteria,  continued 

Procedure  for  a  Gravity  Davit  Lifeboat  ’ 

•  Candidate: 

-  Inspects  releasing  gear 

Part  2:  Develop  Performance  Standards 

-  Checks  releasing  gear  lever 

-  Inspects  tricing  pendant,  sea  painter 

-  Checks  drain  plug 

-  Inspects  general  conditions,  arrangement  oflifeboat  equipment 

-  Inspects  man  ropes,  davit  tracks,  trapping  lines 

-  Locates  hand  crank 

-  Communicates  with  other  crew  members  clearly  using  proper  terminology 
•  After  each  task,  candidate  reports  to  assessor  on  each  measure 

Dev»lopiri|  Manner  A* lewmcnii  lf>3of  1 3" 
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Measures  and  Standards 

Measure 

Standard 

Correct  charging 

99% 

Correct  order  filled 

90% 

CPA 

More  than  0.25  nm 

ARPA  set-up  time 

Less  than  5  minutes 

Drvriopiri  Mariner  A**rMto«m 
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Performance  Standards 


•  Performance  standards  are  the  criteria 
established  as  acceptable  or  target  levels 

-  Individual  measure 

-  Set  of  measures  used  to  assess  proficiency 

•  Performance  standards  incorporate: 

-  Measure  type 

-  Specific  performance  measure 

-  Specific  assessment  conditions 

DMlopms  M«non  A.^-nmutU  u,*°fl  _ _ 
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Performance  Standards  Example 

Developing  Performance  Standards 

•  Specifications  for  equipment  purchase 

•  Identify  candidate  action  or  outcome 

•  Select/define  measure 

-  Determine  requirements  and  objectives 

•  Define  general  standard  strategy 

-  Establish  specifications  (standards) 

•  Get  input  &  reach  consensus  on  standard  for 

-  Purchase  equipment 

each  measure 

-  Check  out  equipment 

*  Set  acceptable  range/develop  precise 

Dfvriopu*  M  nvtr  IWefljn 

definition  for  standards 
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Pass/Fail  vs.  Graded  Standards 


•  Pass/Fail  standards  establish  a  minimum 
level  of  required  ability 

•  Graded  standards  establish  levels  of 
performance 


F}tvrk>piog  M  fcrincr  AiworanW 


Graded  vs.  Pass/Fail  Standards 

Measure  Used  in  Standard 

Standard 

Objective  Rating  Checklist 

Graded 

•  •  • 

Pass/Fail 

• 

Developing  MBriarrAitrstfitnlt 

Developing  Standards  for  a  Single 
Performance  Measure _ 


•  Basis  (graded  vs.  pass/fail)? 

•  Who  establishes  scoring  procedures? 

•  What  is  the  process  for  gaining  agreement 
on  the  standard? 


nrvrlopiiig  Marnier  AmpmoxtiU  M3ofl5*f 


Example  of  Single  Pass/Fail 
Standard _ 


Lookout  Assessment  Objective  4 


pass  fail 


4. 1  Measure:  Answers  "Identify  6  of  the  sightings  that 
should  be  reported  when  detected  by  the  lookout” 

Standard:  Correct  response  includes  6  from  a  list  of 
appropriate  responses 

4.2  Measure:  Reports  sighted  objects  using  ship’s  bell 
Standard:  Reports  at  least  3  surface  objects  using 
ship’s  bell.  Improper  reporting  constitutes  failure. 

Failure  to  detect  visible  objects  within  a  specified 
sector  of  view  constitutes  failure. 

4.3  Measure:  Reports  sighted  objects  verbally  using  degrees 
Standard:  Verbally  reports  at  least  3  surface  objects. 

Reports  must  be  within  22-1/2°  of  actual  bearing. 

(Continued) 
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Example  of  Single  Graded  Standard 


Rules  CPA  Measures  and  Standards  for  Selected 

Minimum  CPA 

Rating 

Greater  than  0.5  nm 

Expert 

Between  0.25  and  0.5  nm 

Qualified 

Less  than  0.25  nm 

Unqualified 

Scoring  Procedures 


•  Combine  multiple  measures  and  standards 

•  Define  proficiency  in  a  selected  area  of 
assessment 


Types  of  Scoring  Procedures 


IF  the  knowledge,  skill,  or  ability 
being  measured  is... 

THEN,  scoring  procedures 
should  be  developed  that  apply... 

essential  to  sate  and/or  effective  job 
performance 

a  pass/fail  approach 

useful  in  safe  and/or  effective  job 
performance,  but  other  mariner 
capabilities  could  readily 
compensate 

a  graded  approach 

Developing  Standards  for 
Combined  Performance  Measures  _ 

•  Identity  measures  &  standards  of  critical 
and  non-critical  job  requirements 

•  Determine  if  measures  should  be  combined 
NO:  Multiple  criteria  for  separate  measures 
YES:  Single  criteria  for  combined  measures 

•  Develop  pass/fail  &  graded  scoring 
procedures 

•  Develop  comprehensive  proficiency 
criterion  across  all  measures 

[VvrfojRnt  M«tmt  >l,<lfl*<> 


Developing  Combined  Standards 
Example:  HamburgerHeaven^_ 

•  Multiple  criteria  for  separate  measures 
-  If  candidate  fails  any  standard,  DO  NOT  HIRE 


Developing  Minever  A**et*n 


Developing  Performance  Standards 


lej  Lookou^ssessmen^bjecti^ 


Measure 

Standard 

Score 

l .  Identifies  six  reportable  sightines 

Correctly  reports  six  of  appropriate 

Pass/ 

sightings  (from  list) 

Fail 

2.  Reports  sighted  objects  using 

Conectly  reports  at  least  3  surface 

Pass ! 

ship’s  bell 

objects  using  ship’s  bell 

3.  Reports  sighted  objects  verbally 
using  degrees 

Correctly  reports  at  least  3  surface 
objects,  within  22-1/2°  of  actual 

Pass/ 

Fail 

bearing 

Pass/ 

4.  Reports  sighted  objects  visually 

Correctly  reports  at  least  3  surface 

using  points 

objects  within  +  2  points  of  actual 

Fail 

bearing 

Pass/ 

Fail 

5.  Reports  sky  objects  verbally 
using  points 

Correctly  reports  at  least  3  sky 
objects  within  +  2  pts.  of  actual 

6.  Reports  audible  targets  verbally 

Conectly  repots  at  least  3  audible 

Pass/ 

using  points 

targets  within  +  2  pts.  of  actual 

Fail 

Combined  Standard:  Candidate  must  pass  each  measure  to  pass  objective, 
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Combined  Standards  for  Lookout 


•  Candidate  must  successfully  perform  each 
action  identified  under  an  assessment 
objective  to  pass  that  objective 

•  Candidate  must  pass  each  of  the  nine 
lookout  assessment  objectives  in  order  to 
pass  the  assessment  and  demonstrate 
proficiency  in  lookout  duties 


Developing  Mnriner  A«te«incHU 


Rules  of  the  Road  Example 
(Review) 

Rules 

^  Prepared  Written 

Knowledge  ■» 

Questions 

Situation 

Mock  Equipment— Highly 

Assessment  M 

Controlled  View  of 

Simulated  Traffic 

Mock  Equipment  —  Control  of 

Collision  m 

Vessel  in  Moderately 

Avoidance  ™ 

Controlled  Simulated  Traffic 

Developing  Mnriner  Autwtnentt 
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Developing  Standards  Example: 
Rules  of  the  Road-1 


•  Collision  Avoidance  Score  (simplified) 


Radar 

Viewing 

Score 


X  (Weight)  + 


Bridge 

Viewing 

Score 


CPA  to 
Threat 
Score 


X  (Weight)  + 


Collision 

Avoidance 

Score 


X 


(Weight)  + 


Dwttopinf  Manner  At* 


1 23  or iso 


Collision  Avoidance  Proficiency 
Standarc^^ — ^ ^ 

•  Maximum  Score  =  20 

•  Standard  for  Proficiency  =10 


Developing  Standards  Example: 

Rule^)frheRoac^_(^Mii(^Mi^— 

•  Collision  Avoidance  +  Situation  Assessment  +  Knowledge 

X  (Weight)  + 

X  (Weight)  + 

X  (Weight)  + 


Collision 

Avoidance 

Score 


Situation 

Assessment 

Score 


Rules 

Knowledge 

Score 


Rules  of  the  Road  Proficiency 
Standarc^ ^ — 

•  Maximum  Score  =  60 

•  Standard  for  Proficiency  =  35 
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Comparing  Simulator  and  Shipboard 
Assessment 

Exercise  6:  Developing 

Performance  Standards 

•  How  would  you  need 
to  develop 

performance  standards 
differently  if  you  were 
assessing  via  simulator 
or  shipboard 
assessment? 

•  Goal:  To  develop  performance  standards 
for  up  to  three  lifeboat  launching 
assessment  objectives 

•  Tools: 

-  Performance  measures  developed  in  Ex.  5 
-Examples 

•  Rules  of  the  Road 

•  Lookout 

Ocvrtop»|  M«wf 

Exercise  6,  continued 

Exercise  6,  continued 

•  To  get  started: 

•  Determine  standard(s)  for  each  measure 

-  Consider  performance  measures  you  developed 

•  Specify  your  rationale 

-  Specify  a  scoring  strategy  for  each  measure 

•  Note  how  you  would  combine  measures  for  a 

•  Pass  /  fail 

single  objective 

•  Graded  (e.g.,  expert,  qualified,  not  qualified) 

•  Discuss  development  of  measures/objectives  into  a 

-  Determine  who  should  be  included  in 

combined  proficiency  criterion 

consensus-gathering  meeting 

•  Record  solution  on  your  transparency 

•  Select  a  spokesperson 

•  You  have  45  minutes 

Drwlopitll  Aiirumrrrtl  t:9  of  1  9' 
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Exercise  6:  Example  Solution 


•  Assume  shipboard  inspection  of  Objective  #1  - 
“person  in  charge  carries  out  proper  inspections” 

•  Use  graded  level  scoring  &  define  levels 

-  Expert:  Measure  accomplished  correctly,  efficiently, 
without  hesitation,  using  proper  terminology 

-  Qualified:  Measure  accomplished  correctly  with  some 
thought  and  hesitation.  Minor  mistakes  in  terminology 
may  be  made. 

-  Not  qualified:  Measure  not  accomplished,  or 
accomplished  incorrectly.  Incorrect  terminology  used, 
which  may  result  in  misunderstandings. 


Exercise  6:  Example  Solution, 

•  Example  performance  measures  &  standards 

-  Measure:  Candidate  properly  inspects,  or  supervises 
the  inspection  of,  releasing  gear. 

-  Standard:  Candidate  correctly  reports  the  condition  of 
the  releasing  gear. 

-  Measure:  Candidate  locates  hand  crank. 

-  Standard:  Candidate  correctly  reports  proper  location 
of  hand  crank. 
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Steps  in  Preparing  Assessment 

Step  5:  Prepare  Assessment 

Materials 

Materials 

1 .  Prepare  detailed  assessment  worksheets 

2.  Prepare  assessor  instructions 

3.  Prepare  candidate  instructions 

4.  Finalize  written  and  oral  test  questions 

5.  Finalize  simulator  and/or  shipboard  procedures 

6.  Document  references,  source  materials,  and 

validation  process 

Drvrlopinf  Mariner  AunintM  IM«fl30 
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Prepare  Documentation 


•  Documentation  provides  a  basis  for  external 
review  &  simplifies  updating  in  the  future 

•  Documentation  should  include: 

-  Source  of  assessment  objectives  plus  mapping  between 
objectives,  conditions,  and  measures 

-  Standards  and  source  of  standards 

-  Detailed  procedures  for  conducting  assessment 

-  Assessment  items/assessor  materials,  candidate 
materials,  and  performance  measurement  procedures 

Dtwlopin,M-i.rrA»«.m«u  IKBfIJO  _ 


ARPA  Assessment  Example: 
A^sessmen^Dbjectives^^ — 

ARPA  Assessment:  Situation  Assessment 

1 .  Understanding  the  criteria  for  the  selection  of  targets  by 
automatic  acquisition 

2.  Appreciation  of  the  uses,  benefits  and  limitations  of 
ARPA  operational  warnings 

3.  Detection  and  identification  of  false  echoes,  sea  return, 
racons,  and  SARTs 

4.  The  use  of  graphic  representation  of  danger  areas 

5.  Knowledge  and  recognition  of  historic  data  as  a  means 

of  indicating  recent  maneuvering  of  targets _ 

Developing  Mwinvr  Aiicwoxflf*  lJJoflW 


ARPA  Assessment  Example: 
Assessmen^ondition^^^^^ ^ 

General  Description:  Fully  functional  ARPA 
in  laboratory  setting  using  several  pre¬ 
designed  operational  exercises 
Description  of  Exercise  A 

-  Open  waters  scenario  with  mix  of  other  vessels 

-  Own  ship  and  Target  E  maneuver 

-  Target  D  reduces  speed 

-  Target  C  is  lost,  resulting  in  lost  target  alarm 
actuation 
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ARPA  Assessment  Example: 

Assessor  Instructions 

ARPA  Assessment  Example: 

Candidate  Materials 

Detailed  Assessor  Instructions  for  Exercise  A 

1.  Before  candidates  arrive,  de-initial ize  ARPA  units  so 
speed  and  compass  settings  are  incorrect. 

2.  Before  getting  underway,  verify  that  each  candidate 
has  correctly  initialized  the  ARPA  unit. 

(more  detail  provided) 

3.  Ask  each  student  to  demonstrate  the  available  display 
presentations  and  display  modes. 

(more  detail  provided) 

Continues  for  1 1  specific  assessor  steps 

Description  of  Exercise  A  Candidate 

Materials 

-  Vessel  data  card 

-  Set-up  instructions 

-  Target  data  forms 

-  ARPA  facsimile  showing  vessel  history  trails 

-  Several  questions  addressing  interpretation  of 
vessel  history  trails 

ARPA  Assessment  Example: 

Measures  and  Standards 

Lookout  Assessment  Example: 
Assessment  Objectives 

•  Performance  Scoring 

-  Standards  for  individual  measures  provided 

•  Score  Weighting 

-  Means  of  summing/combining  scores  (TBD) 

•  Proficiency  Standard 

-  Overall  standard  for  ARPA  proficiency  (TBD) 

1 .  Describe  lookout  duties  &  responsibilities 

2.  Identify  lookout  stations  &  safe  routes  onboard 

3.  Describe  &  identify  international  distress  signals 

4.  Demonstrate  lookout  techniques  &  make  lookout  reports  in  clear 

visibility  during  daylight 

5.  Demonstrate  lookout  techniques  &  make  lookout  reports  in  clear 

visibility  at  night 

6.  Demonstrate  lookout  techniques  &  make  lookout  reports  in 

restricted  visibility  during  daylight  or  at  night 

7.  Demonstrate  the  use  of  lookout  equipment 

8.  Demonstrate  man  overboard  procedures 

9.  Demonstrate  lookout  watch  relief  procedures 

Dtv.lopinj  Marcn*.  AiMMmrt* 
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Lookout  Assessment  Example: 
Assessment  Conditions 

Lookout  Assessment  Example: 

Candidate  &  Assessor  Materials 

Lookout  Assessment  Objective  4: 

Candidate  Materials: 

•  Lookout  station  equipped  with  internal 

-  Candidate  Instructions 

communications  system,  ship’s  bell,  7x50 

-  Assessment  Control  Sheet 

binoculars,  and  bearing  repeater.  Lookout  station 

should  be  clear  and  Assessor  must  be  able  to 

Assessor  Materials: 

observe  activities. 

•  Conduct  in  clear  visibility  during  daylight. 

-  Assessor  Instructions 

•  Assessor  should  ensure  that  reportable  objects  are 

-  Assessment  Worksheets 

in  sight. 

-  Assessment  Control  Sheet 

DrvtlopiniMlriiirr  XlMwntnt*  MJnfIJrt 
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Lookout  Assessment  Example: 


Measures  and  Standards 


Lookout  Assessment  Objective  4 

Measure  Standard 


1.  Identifies  six  reportable 
sightings 

2.  Reports  sighted  objects  using 
ship’s  bell 

.1.  Reports  sighted  objects  verbally 

using  degrees 

4.  Reports  sighted  objects  visually 
using  points 

5.  Reports  sky  objects  verbally 
using  points 

6.  Reports  audible  targets  verbally 

using  points  _ 


Correctly  reports  six  of  appropriate 
sightings  (from  list) 

Correctly  reports  at  least  3  surface 
objects  using  ship’s  bell 
Correctly  reports  at  least  3  surface 
objects,  within  22-1/2°  of  actual 
bearing 

Correctly  reports  at  least  3  surface 
objects  within  +  2  points  of  actual 
bearing 

Correctly  reports  at  least  3  sky 
objects  within  +  2  pts.  of  actual 
Correctly  repots  at  least  3  audible 
targets  within  ±  2  pts.  of  actual 


DrvtlrotcK  M*noer 


Score 

Pass/ 

Fail 

Pass/ 

Fail 

Pass/ 

Fail 

Pass / 
Fail 

Pass/ 

Fail 

Pass/ 

Fail 


Comparison  of  Simulator  and 
Shipboard  Assessment _ 

•  How  would  you  need  to  prepare  the 
assessment  materials  differently  if  you  were 
using  simulation  or  shipboard  assessment? 
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Summary 

EVvrk>pin§  Manner 

Steps  in  Developing  Mariner 
Assessments  _ __ 

1 .  Specify  Assessment  Objectives 

-  Identify  regulatory  requirements 

-  Analyze  job  requirements 

-  Describe  individual  assessment  objectives 

2.  Determine  Assessment  Methods 

-  Identify  alternative  assessment  methods 

-  Review  advantages  &  disadvantages 

-  Define  assessment  methods 

3.  Specify  Assessment  Conditions 

-  Describe  operational  settings  &  scenarios 

-  Specify  written  &  oral  questions 


Steps  in  Developing  Mariner 
Assessments  (continued) 

Concluding  Remarks 

4.  Develop  Proficiency  Criteria 

-  Identify  components  of  assessment  objectives 

•  This  workshop  presents  a  thorough 

-  Develop  measures  &  standards 

-  Develop  scoring  procedures  &  proficiency  criteria 

overview  of  the  assessment  development 

-  Validate  measures,  standards,  procedures  &  criteria 

process. 

5.  Prepare  the  Assessment  Materials 

-  Prepare  detailed  assessment  worksheets 

-  Prepare  assessor  &  candidate  instructions 

•  Accompanying  materials  provide  greater 

-  Finalize  written  &  oral  questions 

detail  and  should  be  referred  to  as  needed. 

-  Finalize  simulator  &/or  shipboard  procedures 

-  Document  references,  source  materials,  &  validation 
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EXERCISE  1 

Understanding  Validity  and  Reliability 


Time: 

Purpose: 

Outcome: 


Report: 


30  minutes 

Practice  reviewing  factors  that  influence  assessment  validity  and  reliability 

A  list  of  positive  and  negative  validity  and  reliability  issues  for  each  assessment 
procedure,  as  well  as  recommended  changes  to  improve  assessment  validity 
and/or  reliability 

Provide  a  review  of  findings  and  recommendations  for  one  assessment  procedure 
(3  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  gain  an  understanding  of  the  important  concepts  of  validity  and 
reliability,  as  they  pertain  to  the  development  of  assessment  procedures. 

Your  tools  for  this  exercise  are: 

□  Lecture  slides  on  validity  and  reliability. 

□  Attachment  1 ,  Hamburger  Limbo  Assessment  Procedures. 

□  Attachment  2,  Validity  and  Reliability  Review  of  Hamburger  Limbo  Assessment 
Procedures. 

B.  To  get  started,  get  in  your  small  groups  and  read  Attachment  1 .  Carefully  consider  all 
aspects  of  the  portion  of  the  Hamburger  Heaven  assessment  that  your  group  is  assigned.  As 
a  small  group,  come  up  with: 

□  Positive  and  negative  validity  issues. 

□  Positive  and  negative  reliability  issues. 

□  Recommended  improvements  for  validity  and  reliability. 

In  conducting  your  review,  focus  on  the  general  assessment  content  and  format  and  consider 
how  these  affect  validity  and  reliability.  Please  DO  NOT  concern  yourselves  with  the 
standards  that  may  be  applied  for  passing  and  failing  each  assessment  (this  will  be  considered 
later). 

Use  Attachment  2  to  develop  a  draft  of  your  ideas;  record  your  group’s  final  answers  on  your 
overhead  transparency  worksheet. 

C.  Select  a  spokesperson  to  present  your  review  to  the  larger  group. 
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EXERCISE  1 
Attachment  1 

Hamburger  Limbo  Assessment  Procedures 


Hamburger  Heaven  employee  candidates  receive  a  full  day  of  training  at  Hamburger  Limbo. 

Candidates  must  successfully  complete  the  following  four-part  Hamburger  Limbo  assessment  in 

order  to  advance  to  a  probationary  position  at  Hamburger  Heaven: 

1.  Computer  Administered  Multiple-choice  Test 

□  Candidates  are  administered  a  multiple-choice  test  using  an  MS  Windows-based  test 
administration  and  scoring  software  program. 

□  The  program  requires  no  keyboard  skills.  Candidates  only  need  to  use  the  cursor 
control  (mouse)  to  select  their  answer  to  each  question  and  scroll  down  the  screen  to 
view  the  additional  questions. 

□  The  multiple-choice  test  items  consist  of  50  questions.  Ten  items  are  included  from 
each  of  the  following  topics: 

1 .  Hamburger  Heaven  mission  and  vision. 

2.  Hamburger  Heaven  profit  goals. 

3.  Worker  safety. 

4.  Server’s  attire  and  hygiene  requirements. 

5.  The  four  standards  of  Hamburger  Heaven  server  performance. 

□  When  each  question  is  answered,  candidates  are  not  allowed  to  change  their  response. 

□  When  all  items  are  completed,  the  test  automatically  closes  and  scores  the  candidate’s 
responses. 

2.  Greeting  Assessment 

□  Candidates  are  seated  in  a  large  room  with  five  counters  at  the  front  of  the  stage. 

□  In  sets  of  five,  candidates  are  instructed  to  stand  behind  the  counter  and  greet  and  take 
orders  from  a  series  of  10  “customers.” 

□  The  10  “customers”  are  actually  professional  actors.  The  “customers”  rotate  through 
the  five  counters  ordering  Hamburger  Heaven  products. 

□  Each  “customer”  is  instructed  to  verbally  abuse  the  candidates  while  giving  their 
order. 

□  An  assessor  is  seated  in  front  of  the  counter  and  records  whether  or  not  the  candidate: 

1 .  Smiles. 

2 .  Greets  the  customer  with  the  company  greeting  “  We  are  happy  to  have  you  at 
Hamburger  Heaven.  May  I  take  your  order  please ?” 

3.  Continues  to  behave  courteously  throughout  the  exchange  with  the 
“customer.” 
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3.  Order  Taking  Assessment 

□  Candidates  are  given  a  brief  orientation  on  the  HH  Integrated  Cash  Register/Order 
Placing/Inventory  Tracking/Performance  Monitoring  System  (the  H-HICROPIRPMS). 

□  After  the  orientation,  candidates  are  asked  to  sit  at  a  table  with  H-HICROPIRPMS 
workstation  and  wear  headphones. 

□  A  series  of  20  orders,  each  with  an  average  of  five  items,  are  provided  over  a  taped 
message  at  a  steady  pace  of  a  15-minute  period. 

□  Candidates  are  required  to  enter  the  orders  in  the  H-HICROPIRPMS. 

□  Candidates  are  assessed  on  the  basis  of  errors  in  the  ordering  of  the  1 00  items. 

4.  Order  Filling  Assessment 

□  Candidates  are  put  behind  the  counter  again  and  handed  a  set  of  20  orders,  each  with 
an  average  of  five  items. 

□  Candidates  are  required  to  fill  all  20  orders,  placing  the  correct  artificial  food  item  on 
a  tray  or  in  a  bag  for  each  order  within  a  15-minute  period. 
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EXERCISE  1 
Attachment  2 

Validity  and  Reliability  Review 
of  Hamburger  Limbo  Assessment  Procedures 


ASSESSMENT  PROCEDURE: 

VALIDITY  ISSUES: 

Assessment  Procedures  Positively  Affecting  Validity 

Assessment  Procedures  Negatively  Affecting  Validity 


RELIABILITY  ISSUES: 

Assessment  Procedures  Positively  Affecting  Reliability 

Assessment  Procedures  Negatively  Affecting  Reliability 


RECOMMENDATIONS  FOR  IMPROVING  THE  ASSESSMENT  PROCEDURES: 
Improvements  to  Validity 

Improvements  to  Reliability 
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EXERCISE  1 

Understanding  Validity  and  Reliability 
Example  Solution 


ASSESSMENT  PROCEDURE:  Greeting  Assessment 
VALIDITY  ISSUES: 

Assessment  Procedures  Positively  Affecting  Validity 

•  See  reaction  to  difficult  customers. 

•  Role  play  behind  counter. 

Assessment  Procedures  Negatively  Affecting  Validity 

•  Candidates  see  other  “customers.” 

•  Candidates  learn  what  to  expect. 

RELIABILITY  ISSUES: 

Assessment  Procedures  Positively  Affecting  Reliability 

•  Dedicated  assessor. 

•  Some  behavior  well-defined. 

Assessment  Procedures  Negatively  Affecting  Reliability 

•  “Smile”  not  well-defined. 

•  “Courteous”  not  well-defined. 

RECOMMENDATIONS  FOR  IMPROVING  THE  ASSESSMENT  PROCEDURES: 

Improvements  to  Validity 

•  Insert  this  into  a  broader  assessment. 

•  Assess  candidates  individually. 

•  Broader  range  of  customer  behavior. 

Improvements  to  Reliability 

•  Define  “smile”  better. 

•  Define  “courteous”  better. 
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Exercise  2 

Specifying  Assessment  Objectives 


Time: 

Purpose: 

Outcome: 

Report: 


30  minutes 

Practice  specifying  assessment  objectives 

List  of  assessment  objectives  for  lifeboat  task 

List  the  objectives  your  group  specified  (3  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  specify  up  to  six  assessment  objectives  for  a  lifeboat  launching 
assessment. 

Your  tools  for  this  exercise  include  the  following  five  attachments: 

1 .  STCW  &  CFR  Lifeboat  Competencies. 

2.  Launching  Procedure  for  a  Gravity  Davit  Lifeboat. 

3.  Identifying  Assessment  Objectives  for  Lookout  Procedures. 

4.  Identifying  Assessment  Objectives  for  ARPA. 

5.  Lifeboat  Launching  Assessment  Objectives  Worksheet. 

To  get  started,  spend  a  few  minutes  reading  Attachments  1  and  2,  the  lifeboat  launching  job 
and  task  data.  Then,  review  the  examples  in  Attachments  3  and  4  and  use  them  as  a  guide 
for  the  format  of  your  lifeboat  launching  assessment  objectives.  Use  Attachment  5,  the 
exercise  worksheet,  to  make  preliminary  notes  about  your  assessment  objectives. 

B.  As  a  small  group,  consider  the  tasks  involved  in  lifeboat  launching  and  specify  up  to  6 
assessment  objectives,  recording  your  group’s  answers  on  the  overhead  transparency 
worksheet. 

Remember,  each  objective  specified  should  describe  what  the  candidate  needs  to  perform 
successfully  to  pass  the  lifeboat  launching  assessment.  Each  objective  should  include: 

□  An  action  (should  relate  to  each  task  you  want  to  observe). 

□  A  set  of  preliminary  conditions  (can  include  any  equipment  you  use  in  the 
assessment,  the  weather,  time,  location,  and  other  variables  -  these  conditions  are 
“preliminary”  because  they  will  be  refined  later  in  the  assessment  development 
process  in  Step  3  -  Specify  Assessment  Conditions ). 

□  A  set  of  preliminary  standards  (can  be  derived  from  STCW  and  CFR  guidelines  and 
best  practices  of  the  marine  industry  -  the  standards  are  “preliminary”  because  they 
will  be  refined  later  in  Step  4  -  Develop  Proficiency  Criteria). 

Example  assessment  objective:  Given  a  simulated,  relatively  complex,  at  sea  scenario  with 
diverse  traffic  (condition),  using  ARPA  calculate  the  required  new  course  for  own  ship  (action) 
to  maintain  a  minimum  CPA  of  2  nm  between  own  ship  and  other  vessels  (standard). 

C.  Select  a  spokesperson  to  present  the  lifeboat  launching  assessment  objectives  specified  by 
your  group. 
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EXERCISE  2 
Attachment  1 

STCW  Code  &  CFR  Lifeboat  Competencies 


STCW  Code  Table  A- VI/2-1:  Specification  of  the  minimum  standard  of  competence  in  survival 
craft  and  rescue  boats  other  than  fast  rescue  boats. 

STCW  Competence:  Take  charge  of  a  survival  craft  or  rescue  boat  during  and  after  launch. 

STCW  Knowledge,  Understanding,  and  Proficiency:  Various  methods  of  devices  used  for 
launching  survival  craft  and  rescue  boats;  methods  of  launching  survival  craft  into  a  rough  sea. 

STCW  Table  A-VI/2-1  Methods  for  Demonstrating  Competence:  Give  correct  commands  for 
launching  and  boarding  survival  craft,  clearing  the  ship  and  handling  and  disembarking  persons 
from  survival  craft.  Prepare  and  safely  launch  survival  craft  and  clear  the  ship’s  side  quickly. 

46  CFR  12.10-  Lifeboatman  §12.10-5  Examination  and  demonstration  of  ability. 

(a)  Before  an  applicant  is  certified  as  a  lifeboatman,  he  or  she  shall  prove  to  the 
satisfaction  of  the  Coast  Guard  by  oral  or  other  means  of  examination,  and  by 
actual  demonstration,  his  or  her  knowledge  of  seamanship  and  the  ability  to  carry 
out  effectively  all  the  duties  that  may  be  required  of  a  lifeboatman.  The  applicant 
shall  demonstrate  that  he  or  she: 

(1)  Has  been  trained  in  all  the  operations  connected  with  the  launching  of 
lifeboats  and  life  rafts,  and  in  the  use  of  oars; 

(2)  Is  acquainted  with  the  practical  handling  of  boats;  and 

(3)  Is  capable  of  taking  command  of  the  boat’s  crew. 

(b)  The  examination,  whether  administered  orally  or  by  other  means,  must  be 
conducted  only  in  the  English  language  and  must  consist  of  questions  regarding: 

(1)  Lifeboats  and  life  rafts,  the  names  of  their  essential  parts,  and  a  description 
of  the  required  equipment; 

(2)  The  clearing  away,  swinging  out,  and  lowering  of  lifeboats  and  life  rafts, 
the  handling  of  lifeboats  under  oars  and  sails,  including  questions  relative 
to  the  proper  handling  of  a  boat  in  heavy  sea;  and 

(3)  The  operation  and  functions  of  commonly  used  methods  of  davits. 

(c)  The  practical  examination  shall  consist  of  a  demonstration  of  the  applicant’s 
ability  to  carry  out  the  orders  incident  to  launching  lifeboats,  and  the  use  of  the 
boat’s  sail,  and  to  row. 
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EXERCISE  2 
Attachment  2 

Launching  Procedure  for  a  Gravity  Davit  Lifeboat 


( 1 )  Boat  captain  inspects  boat  and  davits  prior  to  deployment. 

□  Check  for  potential  fouls  in  lines,  ladder. 

□  Check  davit  trackways. 

□  Remove  cover  /  strongbacks  as  required. 

(2)  Release  the  man  ropes. 

□  Check  the  tricing  pendants. 

(3)  Put  the  boat  plug  in. 

□  Check  releasing  gear  lever -lever  down,  cotter  pin  secured. 

(4)  Rig  the  sea  painter. 

□  “Rig  the  sea  painter.” 

□  Inboard  of  the  falls,  outboard  of  all  else  on  ship. 

(5)  Release  the  gripes. 

□  “Release  the  gripes.” 

□  “Heads  up.” 

□  Lower  to  secure  position. 

(6)  Lower  to  embarkation  deck. 

□  “Lower  away.” 

□  Davits  should  move  down  trackways  together. 

(7)  Rig  trapping  lines. 

□  “Pass  and  secure  frapping  lines.” 

□  Secure  with  a  strain. 

(8)  Embark  passengers. 

□  Keep  center  of  gravity  low;  hold  man  ropes. 

(9)  Trip  the  tricing  pendants  simultaneously. 

□  “Release  the  tricing  pendants.” 

□  Use  trip  line  or  hatchet;  keep  clear  of  swing  of  hooks  as  released. 

(10)  Tend  trapping  lines. 

□  “Ease  the  trapping  lines.” 

□  Slack  until  falls  vertical. 

(11)  Lower  boat  to  water. 

□  “Lower  away.” 

□  Fend  off  using  boat  hooks. 

(12)  Release  falls. 

□  “Release  falls.” 

□  Release  at  the  crest  of  the  wave. 

(13)  Clear  vessel. 

□  Pass  fall  blocks. 

□  Secure  man  ropes  as  required  to  avoid  fouling  screw. 

(14)  Release  sea  painter. 

□  If  releasing  sea  painter  at  falls,  secure  bitter  end  to  fall  clear  of  water. 
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EXERCISE  2 
Attachment  3 

Example:  Specifying  Assessment  Objectives 
For  Performance  of  Lookout  Procedures 


STCW  Code  Data 

STCW  Table  A-ll/4:  Mandatory  minimum  requirements  for  certification  of  ratings  forming  part 
of  a  navigational  watch. 

STCW  Function:  Navigation  at  the  support  level. 

STCW  Competence:  Keep  a  proper  lookout  by  sight  and  hearing. 

STCW  Proficiency:  Responsibilities  of  a  lookout,  including  reporting  the  approximate  bearing 
of  a  sound  signal,  light,  or  other  object  in  degrees  or  points. 

Additional  Task  Data 

Although  it  is  not  specifically  addressed  in  the  regulations,  it  is  important  that  the  mariner 
(1)  understand  the  lookout  duties  and  procedures,  particularly  as  they  affect  safety  on  the  vessel; 
and  (2)  demonstrate  lookout  techniques.  These  requirements  may  be  summarized  in  the 
following  five  areas: 

•  Knowledge  of  lookout  duties  and  responsibilities. 

•  Knowledge  of  lookout-related  safety  procedures. 

•  Demonstration  of  lookout  procedures,  under  various  conditions. 

•  Demonstration  of  lookout  equipment. 

•  Demonstration  of  related  procedures,  such  as  man  overboard  and  lookout  watch  relief. 
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Example  Objectives  for  Lookout  Procedures  Assessment 

Example  assessment  objectives  were  derived  using  the  information  above  as  well  as  consultation 
with  subject  matter  experts  (SMEs): 

Assessment  Objective  1 

Action:  Describe  lookout  duties  and  responsibilities. 

Preliminary  Conditions:  Examination  proctor,  test  instrument,  and  any  associated  references. 
Preliminary  Standards:  Mariner  specifies  lookout’s  duties  as: 

Sighting,  identifying,  and  accurately  reporting  all  objects  or  sounds  detected. 

Assessment  Objective  2 

Action:  Identify  lookout  stations  and  safe  routes  onboard. 

Preliminary  Conditions:  Examination  proctor,  test  instrument,  and  any  associated  references. 

Preliminary  Standards:  Mariner  identifies  such  common  lookouts,  and  the  following  relevant 
safe  routes,  of: 

1 .  Bridge. 

2.  Bridge  wings. 

3.  Bow. 

Assessment  Objective  3 

Action:  Describe  and  identify  international  distress  signals. 

Preliminary  Conditions:  Examination  proctor,  test  instrument,  and  any  associated  references. 

Preliminary  Standards:  Mariner  identifies  and  describes  some  of  the  following  international 
distress  signals: 

•  Red  star  shells. 

•  Fog  horn  continually  sounding. 

•  Flames  or  smoke  on  a  vessel. 

•  Gun  fired  at  intervals  of  1  minute. 

•  SOS. 

•  “Mayday”  by  radio. 

•  Parachute  red  flare. 

•  Wave  arms. 

•  Etc. 

Assessment  Objectives  4-6 

Action:  Demonstrate  lookout  techniques  and  make  lookout  reports. 

Preliminary  Conditions:  Clear  lookout  station  properly  equipped;  reportable  objects  in  sight; 
conditions  of  1)  Daylight,  clear  visibility,  2)  night,  clear  visibility,  3)  restricted  visibility  (day  or 
night). 
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Preliminary  Standards:  Mariner  correctly  performs  the  following  actions: 

1 .  Identifies  six  of  the  relevant  and  appropriate  sightings  that  should  be  reported  when 
detected  by  the  lookout. 

2.  Reports  sighted  surface  objects  using  ship’s  bell. 

3.  Reports  sighted  surface  objects  verbally  using  a)  degrees  and  b)  points. 

4.  Reports  sighted  sky  objects  verbally  using  points. 

5.  Reports  audible  targets  verbally  using  points. 

Assessment  Objective  7 

Action:  Demonstrate  the  use  of  lookout  equipment. 

Preliminary  Conditions:  Clear  lookout  station  properly;  reportable  objects  in  sight;  any  of  the 
following  conditions  of  visibility:  1)  Daylight,  clear  visibility,  2)  night,  clear  visibility,  3) 
restricted  visibility  (day  or  night). 

Preliminary  Standards:  Mariner  correctly  identifies  and  demonstrates  use  of  the  following 
equipment: 

1.  Ship’s  bell. 

2.  Internal  communications  system. 

3.  Binoculars. 

4.  Bearing  repeater  fitted  with  bearing/azimuth  circle,  alidade,  or  pelorus. 

5.  Personal  equipment. 

Assessment  Objective  8 

Action:  Demonstrate  man  overboard  procedures. 

Preliminary  Conditions:  Clear  lookout  station  properly  equipped;  reportable  objects  in  sight; 
any  of  the  following  conditions  of  visibility:  1)  Daylight,  clear  visibility,  2)  night,  clear  visibility, 
3)  restricted  visibility  (day  or  night);  assessor  simulates  man  overboard  to  starboard  or  port. 

Preliminary  Standards:  Mariner  correctly  performs  the  following  tasks: 

1 .  Throwing  of  ring  life  buoy. 

2.  Sounding  of  alarm/reporting  man  overboard  to  the  bridge. 

3.  Keeps  the  victim  in  sight  at  all  times. 

Assessment  Objective  9 

Action:  Demonstrate  lookout  watch  relief  procedures. 

Preliminary  Conditions:  Clear  lookout  station  properly  equipped;  reportable  objects  in  sight; 
any  of  the  following  conditions  of  visibility:  1)  Daylight,  clear  visibility,  2)  night,  clear  visibility, 
3)  restricted  visibility  (day  or  night). 

Preliminary  Standards:  Mariner  correctly  answers  the  following  questions: 

1 .  When  may  a  lookout  leave  the  lookout  station  before  being  relieved? 

2.  What  must  be  considered  before  handing  over  the  watch  to  the  watch  relief  during 
darkness? 

3 .  What  constitutes  and  adequate  watch  relief  report? 

4.  What  action(s)  must  be  taken  if  not  properly  relieved? 
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EXERCISE  2 
Attachment  4 

Example:  Specifying  Assessment  Objectives  for  ARPA1 

In  an  example  ARPA  assessment,  McCallum,  et  al.  (1999)  divides  ARPA  knowledge  and  skill 
into  six  areas.  Below  are  the  assessment  objectives  for  the  third  area,  “knowledge  of  factors 
affecting  performance  and  accuracy;  and  ability  to  operate  and  interpret  system  performance  and 
accuracy,  tracking  capabilities  and  limitations,  and  processing  delays.”  The  rest  of  the  example 
ARPA  assessment  objectives  can  be  found  in  Appendix  A  of  the  report,  Developing 
Performance-based  Assessments  of  Mariner  Proficiency. 


Assessment  Objective 

STCW  Reference 

Assessment  Conditions 

Performance 

Measure 

Knowledge  of  effects  of 
limitations  of  radar  range  and 
bearing  on  the  accuracy  of 
ARPA  data 

24.1 

Sect.  B-I/12 

Exercise  F 

Three  vessels  on  same 
initial  bearing,  with  two  on 
collision  course;  and  fourth 
vessel  on  a  different  bearing 

Instructions  to  report  valid 
target  data  ASAP 

3.3.1.  Reported  vessel 
data 

The  circumstances  causing 
“target  swap”  and  its  effects 
on  display  data 

25.4 

Sect.  B-I/12 

Exercise  C 

Vessels  in  open  water 
passing  one  another  and 
buoys 

3.2.1  Identification  of 
pairs  of  targets 
where  swap  is 
likely  to  occur 

The  effects  on  tracking  of 
“lost”  targets  and  target  fading 

25.3 

Sect.  B-I/12 

Exercise  A 

Loss  of  target  track  and 
sounding  of  “target  lost” 
alarm 

3.3.1  Identification  of 
lost  target  alarm 

3.3.2  Reacquisition  of 
lost  target 

An  appreciation  of  the  IMO 
performance  standards  for 
ARPA,  in  particular  the 
standards  relating  to  accuracy 

22.0 

Sect.  B-I/12 

Written  test 

(administered  at  the  end  of 
Exercise  F) 

3.4.1  Correct 

responses  to 
written  test 

1  Excerpted  from  Appendix  A  of  McCallum,  M.  C.,  Forsythe,  A.  M.,  Smith,  M.W  Nunnenkamp,  J.  M.,  &  Sandberg, 
G.  R..  (1999).  Developing  performance-based  assessments  of  mariner  proficiency.  (Report  Number  R&DC-202- 
99).  Groton,  CT:  USCG  R&D  Center. 


A  Method  for  Developing  Mariner  Assessments 


2-43 


Chapter  2-  A  Workshop  for  Assessment  Developers 


EXERCISE  2 
Attachment  5 

Lifeboat  Launching  Assessment  Objectives  Worksheet 


ASSESSMENT  OBJECTIVE  1 
Action: 

Preliminary  Condition(s): 
Preliminary  Standard(s): 

[ASSESSMENT  OBJECTIVE  2 
Action: 

Preliminary  Condition(s): 
Preliminary  Standard(s): 

ASSESSMENT  OBJECTIVE  3 
Action: 

Preliminary  Conditions: 
Preliminary  Standards: 
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ASSESSMENT  OBJECTIVE  4 
Action: 

Preliminary  Condition(s): 
Preliminary  Standard(s): 

ASSESSMENT  OBJECTIVE  5 
Action: 

Preliminary  Condition(s): 
Preliminary  Standard(s): 

ASSESSMENT  OBJECTIVE  6 
Action: 

Preliminary  Conditions: 
Preliminary  Standards: 
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EXERCISE  2 

Specifying  Assessment  Objectives 
Example  Solution 


Assessment  Objective  1 

Action:  Person  in  charge  carries  out  proper  inspections. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor  can 
observe  activities.  The  following  conditions  can  be  varied: 

1 .  The  sea  painter  is  fouled,  missing  or  not  made  fast  properly. 

2.  One  or  both  tricing  pendants  are  not  secured  or  are  secured  improperly. 

3.  The  man  ropes  are  fouled. 

4.  The  drain  plug  cap  is  missing  or  damaged. 

5.  Rudder,  tiller  or  other  equipment  is  missing  or  damaged. 

6.  The  trapping  lines  are  fouled  or  missing. 

Preliminary  standards:  The  person  in  charge  inspects  the  lifeboat,  falls  and  davit.  Using  proper 
commands  and  nautical  terminology  he/she  demonstrates  consideration  and  supervises  the  proper 
performance  of  the  following  steps: 

1 .  Insures  the  lifeboat  releasing  gear  is  attached  properly  to  the  falls. 

2.  Checks  to  make  sure  the  releasing  gear  lever  is  in  the  proper  position  (lever  down),  with 
the  pin  in  place. 

3.  Insures  the  tricing  pendent  is  in  place  and  secured  properly. 

4.  Insures  that  the  sea  painter  is  clear,  attached  to  the  thwart  properly  and  ready  to  lead  out. 

5.  Checks  the  drain  plug  cap  is  in  good  condition  and  ready  for  proper  placement. 

6.  Checks  general  condition  and  arrangement  of  boat  equipment. 

7.  Determines  that  the  man  ropes  are  not  fouled. 

8.  Insures  the  davit  tracks  are  clear. 

9.  Makes  sure  the  trapping  lines  are  in  place  and  ready  to  be  passed. 

1 0.  Insures  that  the  hand  crank  is  in  the  stowed  position. 

Assessment  Objective  2 

Action:  The  person  in  charge  prepares  the  lifeboat  for  launching. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor 
can  observe  activities.  The  following  conditions  can  be  varied: 

Sea  painter  is  secured  improperly  to  the  thwart  or  not  secured  at  all. 

One  or  both  tricing  pendants  are  secured  to  the  falls  improperly  or  not  at  all. 

Preliminary  standards:  The  person  in  charge  prepares  the  boat  for  lowering.  Using  the  proper 
commands  and  nautical  terminology  he/she  demonstrates  consideration  and  supervises  the  proper 
performance  of  the  following  steps: 
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1 .  Puts  the  drain  plug  cap  on. 

2.  Leads  out  the  sea  painter,  forward,  inboard  of  the  falls  outboard  of  everything  else  on  the 
vessel.  Secures  the  painter  to  the  thwart  if  necessary. 

3.  Lowers  man  ropes. 

4.  Makes  tricing  pendants  fast  if  necessary. 

5.  Releases  the  gripes  insuring  that  all  personnel  are  clear  and  out  of  the  lifeboat. 

6.  Removes  gripes  from  the  lifeboat. 

7.  Insures  that  the  locking  bars  are  clear  of  the  davit  tracks. 

Assessment  Objective  3 

Action:  The  person  in  charge  lowers  the  lifeboat  to  the  embarkation  deck. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor  can 
observe  activities. 

Preliminary  standards:  The  person  in  charge  lowers  the  boat  to  the  embarkation  deck.  Using 
the  proper  commands  and  nautical  terminology  he/she  demonstrates  consideration  and  supervises 
the  proper  performance  of  the  following  steps: 

1 .  Clears  all  personnel  from  the  area  around  and  under  the  lifeboat. 

2.  Insures  no  one  has  his/her  hands  on  the  davit  tracks. 

3.  Raises  the  brake  handle  lowering  the  boat  slowly. 

4.  Lowers  the  boat  to  the  embarkation  deck.  The  tricing  lines  bringing  the  boat  to  the  side  of 
the  ship. 

5.  Insures  that  the  lifeboat  does  not  slam  into  the  side  of  the  ship  due  to  lowering  it  too  fast 
or  too  far. 

Assessment  Objective  4 

Action:  The  person  in  charge  embarks  the  crew  and  passengers. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor  can 
observe  activities. 

Preliminary  standards:  The  person  in  charge  lowers  the  boat  to  the  embarkation  deck.  Using 
the  proper  commands  and  nautical  terminology  he/she  demonstrates  consideration  and  supervises 
the  proper  performance  of  the  following  steps: 

1 .  Passes  and  secures  the  frapping  lines  properly.  They  are  secured  under  a  strain  with  no 
slack.  They  must  be  secured  so  that  they  can  be  slacked  without  jamming. 

2.  Insures  that  all  crew  and  passengers  are  wearing  lifejackets  properly. 

3.  Boards  the  crew  and  passengers  in  lifeboat. 

4.  Insures  that  all  personnel  are  seated  and  holding  on  to  a  man  rope. 

5.  Releases  the  tricing  pendants  using  the  tripping  lines  insuring  that  all  personnel  are  clear 
of  the  hook  and  pendant  as  it  swings  clear. 

6.  Slacks  the  frapping  lines  easing  the  lifeboat  outboard  until  the  falls  are  vertical. 
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Assessment  Objective  5 

Action:  The  person  in  charge  lowers  the  lifeboat  to  the  water. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor  can 
observe  activities. 

Preliminary  standards:  The  person  in  charge  lowers  the  boat  to  the  water.  Using  the  proper 
commands  and  nautical  terminology  he/she  demonstrates  consideration  and  supervises  the  proper 
performance  of  the  following  steps: 

1 .  Raises  the  brake  handle  lowering  the  lifeboat  to  the  water’s  edge. 

2.  Insures  no  crew  or  passengers  are  fouled  in  man  ropes. 

3 .  Fends  the  lifeboat  off  from  the  side  of  the  ship  if  necessary. 

Assessment  Objective  6 

Action:  The  person  in  charge  of  the  boat  clears  the  lifeboat  away  from  the  ship. 

Conditions:  Lifeboat  is  available  for  assessment  purposes.  The  boat  is  clear  and  the  assessor  can 
observe  activities. 

Preliminary  standards:  The  person  in  charge  of  the  boat  clears  the  boat  away  from  the  ship. 
Using  the  proper  commands  and  nautical  terminology  he/she  demonstrates  consideration  and 
supervises  the  proper  performance  of  the  following  steps: 

1 .  Releases  the  falls  taking  care  that  no  personnel  are  knocked  overboard  or  injured  by  the 
blocks  or  falls. 

2.  Fends  the  boat  away  from  the  ship  if  necessary. 

3.  Releases  the  sea  painter. 
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EXERCISE  3 

Determining  Assessment  Methods 


Time: 

Purpose: 

Outcome: 

Report: 


1 5  minutes 

Practice  determining  appropriate  assessment  methods 

Appropriate  assessment  methods  for  six  lifeboat  assessment  objectives 

Explain  which  assessment  method(s)  you  selected  for  each  objective  and  why 
(2  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  select  the  most  appropriate  assessment  method  for  each  of  your 
lifeboat  launching  assessment  objectives.  Remember  that  the  basic  assessment  methods  are: 

□  Written  or  oral  questions. 

□  Simulation  using  mock  equipment  (e.g.,  a  personal  computer  interface  in  lieu  of  an 
actual  radar  screen.). 

□  Simulation  in  a  laboratory  using  actual  equipment. 

□  Shipboard  assessments. 

For  a  comparison  of  these  assessment  methods,  see  Attachment  1,  Pros  and  Cons  of  Basic 
Assessment  Methods.  For  an  example  of  the  considerations  involved  in  selecting  assessment 
methods,  refer  to  Attachment  2,  Determining  Assessment  Methods  for  Lookout  Procedures. 
Use  Attachment  3,  Lifeboat  Launching  Assessment  Methods  Worksheet ,  to  make  notes  as  you 
consider  which  assessment  method  to  use  for  each  objective. 

B.  To  determine  the  appropriate  assessment  method  for  each  lifeboat  launching  objective, 
discuss  the  following  questions  in  your  small  group: 

□  What  performance  must  be  assessed? 

-  Review  the  assessment  objectives  specified  in  Exercise  2. 

-  Consider  how  the  task  is  initiated. 

-  Consider  what  the  result  of  task  performance  is. 

-  Consider  the  process  of  performing  the  task. 

□  Review  realism  (validity)  requirements. 

-  Environment/setting. 

-  Controls. 

-  Displays. 

-  Processing  characteristics. 

□  Review  control  (reliability)  requirements. 

-  Key  skill  and  knowledge  requirements. 

-  Variability  of  task  conditions. 

-  Comparability  of  assessment  conditions. 
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□  What  assessment  methods  are  feasible ? 

-  Can  the  performance  be  measured  via  written  assessment? 

-  Can  a  simulator  or  laboratory  replicate  the  conditions  closely  enough  to  cue 
the  behavior? 

-  Can  the  performance  be  easily  and  safely  elicited  onboard  ship? 

-  Which  method  is  the  easiest,  cheapest,  safest,  and  most  legally  defensible? 

NOTE:  If  you  select  shipboard  assessment  as  the  method  for  any  objective,  specify  whether  the 
assessment  should  be  underway  or  in  port. 

C.  Record  your  group’s  answers  on  an  overhead  transparency  worksheet,  and  select  a 
spokesperson  to  present  your  assessment  methods  and  rationale  to  the  larger  group. 
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EXERCISE  3 
Attachment  1 

Pros  and  Cons  of  Basic  Assessment  Methods 


Assessment  Method 

Pros 

Cons 

Written  or  oral 

-  Easy  to  create 

-  May  have  poor  similarity 

Easy  to  standardize 

to  work  environment 

Easy  to  test  many  at  once 

-  Good  for  measuring 
knowledge 

-  Candidate  needs  good 
verbal  skills 

Simulation  using  mock 

-  High  control 

-  May  be  expensive 

equipment 

-  May  have  poor  similarity 
to  work  environment 

-  Usually  need  one  assessor 
per  candidate 

-  Safe 

Simulation  in  a  laboratory 

-  High  control 

-  May  be  expensive 

using  actual  shipboard 

Usually  need  one  assessor 
per  candidate 

equipment 

-  Safe 

-  May  have  high  similarity 
to  work  environment 

Shipboard 

-  May  be  inexpensive 

Uses  actual  work  setting 

-  Usually  need  one  assessor 
per  candidate 

-  May  be  difficult  to  fit  into 
operational  schedule 

-  More  difficult  to 
standardize  and  score 

-  Task  may  be  infrequent  or 
dangerous 
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EXERCISE  3 
Attachment  2 

Example:  Determining  Assessment  Methods 
for  Lookout  Procedures 


Additional  Considerations  in  Assessment  Method  Selection: 

1 .  Assessment  Requirements. 

•  Knowledge  of  lookout  duties  and  procedures. 

•  Correct  use  and  operation  of  lookout  techniques,  reporting,  and  equipment. 

2.  Assessment  Constraints. 

«  Must  not  interfere  with  vessel  safety. 

•  Equipment  must  be  available  and  operational. 

•  Scheduling  to  allow  for  conditions  of  clear  visibility  in  daylight  and  at  night,  plus 
restricted  visibility  during  day  or  night. 

Selection  of  Assessment  Method: 

Written  or  oral  test  questions  are  needed  to  assess  knowledge  of  lookout  duties  and  procedures, 
covering  Assessment  Objectives  1  to  3. 

Correct  use  and  operation  of  lookout  techniques,  reporting,  and  equipment  requires  shipboard  or 
simulator  assessment.  Shipboard  assessment  offers  the  highest  degree  of  validity,  making  a  very 
good  choice  of  assessment  method  provided  that  the  constraints  discussed  above  are  met. 
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EXERCISE  3 
Attachment  3 

Lifeboat  Launching  Assessment  Methods  Worksheet 


Assessment  Objective 

Assessment  Method 

1 .  Carry  out  proper  inspections 

2.  Prepare  lifeboat  for  launching 

3.  Lower  lifeboat  to  the  embarkation 
deck 

4.  Embark  crew  and  passengers 

5.  Lower  lifeboat  to  the  water 

6.  Clear  lifeboat  away  from  the  ship 
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EXERCISE  3 

Determining  Assessment  Methods 
Example  Solution 

Considerations  in  assessment  method  selection 

1 .  Is  written,  knowledge-based  assessment  appropriate?  Is  it  sufficient? 

2.  Availability  of  lifeboat  and  davit. 

3.  Operational  safety. 

4.  Possibility  of  damaging  a  vessel’s  lifesaving  equipment. 

5.  Ability  to  vary  conditions  on  equipment  to  be  used. 

6.  Alternate  methods  available,  such  as  lifeboat  models. 

Discussion  of  considerations 

1.  A  written  or  oral  assessment: 

a)  Would  be  useful  in  determining  a  candidate’s  knowledge  of  lifeboat  launching 
procedures,  terminology,  and  regulations. 

b)  Is  not  a  measure  of  a  candidate’s  ability  to  either  carry  out  or  perform  the  tasks  needed  to 
be  done  in  launching  the  boat. 

c)  Has  poor  face  validity. 

d)  Offers  a  high  degree  of  reliability. 

2.  Availability  of  lifeboats  is  high.  All  vessels  are  equipped  with  them.  In  addition,  most 
training  facilities  have  available  for  training  a  lifeboat  and  davit  either  installed  on  a  pier  or 
aboard  a  training  vessel. 

3.  Operational  safety  must  not  be  compromised  and  would  always  be  a  primary  concern. 
Assessors  must  be  extremely  careful  to  guard  against  candidates  falling  overboard  or  being 
injured  by  equipment.  For  this  reason,  not  all  objectives  can  be  carried  out  while  the  vessel 
is  underway.  It  may  also  be  necessary  for  the  assessor  to  step  in  and  stop  or  correct  a 
procedure  before  an  accident  occurs. 

4.  Damaging  the  lifesaving  equipment  aboard  a  vessel  is  a  concern.  Damaged  equipment  could 
put  the  ship's  crew  at  risk  or  cause  delays  in  port  while  equipment  is  repaired  or  replaced. 
This  would  not  be  as  critical  if  the  assessment  were  carried  out  at  a  training  institution  where 
equipment  is  dedicated  for  training,  not  lifesaving. 

5.  The  ability  to  vary  conditions  of  the  assessment  would  be  the  same  for  lifeboats  aboard  an 
operating  vessel  or  lifeboats  at  a  training  institution.  There  would  be  a  natural  reluctance  for 
a  vessel’s  crew  to  alter  conditions  in  a  lifeboat. 

6.  Model  lifeboats  can  be  used  to  assess  lifeboat  competency.  However,  they  have  poor  face 
validity  and  most  mariners  consider  them  a  poor  substitute  for  a  real  lifeboat. 

Example  assessment  method 

1 .  Lifeboats  aboard  an  operating  vessel  or  at  a  training  institution  could  be  used. 

2.  If  a  lifeboat  aboard  a  vessel  was  used  only  objectives  1-3  could  be  assessed  at  sea  while 
the  vessel  was  underway.  Objectives  4-6  would  have  to  be  assessed  when  the  vessel 
was  in  port.  At  a  training  institution,  all  objectives  could  be  assessed  at  one  time. 
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EXERCISE  4 

Specifying  Assessment  Conditions 


Time: 

Purpose: 

Outcome: 

Report: 


20  minutes 

Practice  specifying  assessment  conditions 

Assessment  conditions  for  one  lifeboat  assessment  objective 

List  the  conditions  your  group  specified  (2  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  specify  assessment  conditions  for  one  lifeboat  launching 
assessment  objective.  Each  group  is  assigned  a  different  objective;  see  your  group’s 
overhead  transparency  worksheet  for  the  objective  that  you  have  been  assigned. 

B.  To  get  started,  refer  back  to  the  preliminary  conditions  that  you  specified  for  this  objective  in 
Exercise  2,  and  the  assessment  method  that  you  specified  for  this  objective  in  Exercise  3. 
Consider  these  as  you  specify  the  conditions  for  this  exercise. 

Attachments  1  and  2  provide  you  with  some  examples  of  assessment  conditions  for  the  lookout 
procedures  assessment  and  ARPA  assessment.  Use  these  examples  as  a  guide  for  the  format  of 
the  assessment  conditions  you  specify  for  your  lifeboat  launching  assessment. 

Remember  that  your  conditions  should  include: 

□  Candidate  orientation. 

□  Equipment,  apparatus,  and  tools. 

□  Initial  equipment  settings  or  scenarios. 

□  Written  or  oral  questions. 

C.  Use  Attachment  3,  Lifeboat  Launching  Assessment  Conditions  Worksheet,  to  do  a 
preliminary  outline  of  the  assessment  conditions  for  your  assessment  objective.  Record  your 
final  answers  on  your  overhead  transparency  worksheet. 

D.  Select  a  spokesperson  to  tell  the  larger  group  the  assessment  conditions  your  group  specified 
and  your  rationale. 
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EXERCISE  4 
Attachment  1 

Example:  Specifying  Assessment  Conditions  for 
Lookout  Procedures 


Discussion  of  requirements  and  constraints  for  Assessment  Objective  4 

1 .  Assume  shipboard  assessment  and/or  ship  bridge  simulator  assessment,  oral  or  written 
test  questions. 

2.  Review  the  preliminary  conditions  under  Assessment  Objective  4. 

3.  Ensure  that  equipment  is  available  and  operational. 

4.  Ensure  safe  operation  of  vessel  and  proper  safety  procedures. 

Assessment  Objective  Measured: 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  during  daylight. 

Candidate  Orientation: 

□  Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Equipment,  Apparatus,  and  Tools: 

□  The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and  bearing 
repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The  lookout  station 
should  be  clear,  and  the  assessor  must  be  able  to  observe  activities. 

Initial  Equipment  Settings  or  Scenarios: 

□  The  assessment  should  be  conducted  in  clear  visibility  during  daylight.  The  assessor 
should  ensure  that  there  are  reportable  objects  in  sight. 

Written  or  Oral  Questions: 

□  Identify  six  of  the  sightings  that  should  be  reported  when  detected  by  the  lookout. 


A  Method  for  Developing  Mariner  Assessments 


2-56 


Chapter  2  -  A  Workshop  for  Assessment  Developers 


EXERCISE  4 
Attachment  2 

Example:  Specifying  Assessment  Conditions  for  ARPA 


The  following  exercise  is  one  of  seven  exercises  comprising  the  example  ARPA  assessment  in 
Developing  Performance-based  Assessments  of  Mariner  Proficiency  (McCallum  et  al.,  1 999).  The 
other  exercises,  candidate  worksheets,  and  assessor  instructions  can  be  found  in  appendices  B,  C,  and 
D  of  that  report. 

Assessment  Objective  Measured: 

Understanding  of  when  to  use  ground-  or  sea-stabilized  modes,  and  when  to  use  north-up, 
course-up,  and  head-up  presentations. 

Exercise  E 
Candidate  Orientation 

•  Assessment  is  conducted  using  a  full-scale  functional  ARPA  in  a  laboratory  setting. 

•  Narrow  channel  navigation. 

•  Cross  current  and/or  wind  requiring  “crabbing”  of  ship  down  the  channel;  crabbing  is 
facilitated  by  ground  stabilized  presentation  method. 

•  Own  ship  is  outbound. 

•  One  threat  vessel  is  inbound. 

Equipment,  Apparatus,  and  Tools 

1 .  Full-scale  ARPA  Simulator. 

2.  ARPA  Set-Up  Instructions. 

a.  Initialize  your  ARPA  unit  with  a  ground-stabilized  presentation  method. 

b.  Set  autodrift  (groundlock)  to  target  bearing  172°  at  4  nm. 

c.  Avoid  collisions  or  groundings. 

3.  Target  Information  Forms. 

a.  Record  vessel  data  for  any  inbound  target(s),  and  include  the  time  you  recorded 
the  data  for  each  target: 


Initial  Bearing 

Target 

Course 

Speed 

CPA 

TCPA 

Time 

A 

B 
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EXERCISE  4 
Attachment  2  (cont.) 


Initial  Equipment  Settings  or  Scenarios: 


1 .  Vessel  data  at  1 1 00:00  (approximate). 


Target 

Bearing 

Range 

Course 

Speed 

CPA 

Own  ship 

NA 

NA 

180 

15 

NA 

NA 

A 

167 

7.0 

347 

11 

0.2 

17.5 

2.  Detailed  scenario  description. 


Time 

Phase 

Event 

Objective 

Number 

Performance 

Measure 

Correct 

Response 

1100:00 

Operator  is  given  a  chartlet 

NA 

NA 

|  NA 

1100:00 

Operator  is  instructed  to  use  ground- 
stabilized  presentation  method 

LI 

Measured  in 
Exercise  A 

See 

Exercise  A 

1101:00 

Simulation 

Operator  instructed  to  set  autodrift  to 
target  bearing  172°  at  4  nm 

1.3 

Measured  in 
Exercise  A 

See 

Exercise  A 

1101:00 

Simulation 

Operator  navigates  own  ship  through 
channel  avoiding  buoys,  land,  and  other 
target 

1.6 

1.6.1 

Maintenance 
of  ship 
position  and 
course 

Bearing  of 
180° 

maintained 
with  no 
groundings 

1102:00 

Simulation 

Operator  instructed  to  record  vessel  data 
for  any  inbound  target(s) 

2.6 

Measured  in 
Exercises  A, 

C,  D 

See 

Exercises 

A,  C,  D 

1113:00 

Simulation 

After  passing  under  bridge,  operator 
reports  lights  that  would  be  seen  on 
Target^  from  1113:00  to  1115:00 

1.6 

|  1.6.2 

Determination 
of  aspect  of 
other  vessel 

Green 

lights 

Written  or  oral  question  (one): 

Your  vessel  is  under  your  control:  What  color  lights  should  you  see  on  the  inbound  target  when 
you  emerge  from  under  the  Verrazano  Bridge,  at  about  1 1 13:00? 
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EXERCISE  4 
Attachment  3 

Lifeboat  Launching  Assessment  Conditions  Worksheet 


Assessment  Objective: 


Assessment  Method: 


Candidate  Orientation: 


Equipment,  Apparatus,  and  Tools: 


Initial  Equipment  Settings  or  Scenarios: 


Written  or  Oral  Questions: 
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EXERCISE  4 

Specifying  Assessment  Conditions 
Example  Solution 

Discussion  of  assessment  requirements  and  constraints 

1.  Assume  shipboard  assessment  of  Assessment  Objective  1,  “person  in  charge  carries  out 
proper  inspections.” 

2.  Review  the  conditions  under  Objective  1  and  determine  if  any  of  the  conditions  should  be 
modified  to  assess  the  candidate’s  performance. 

3.  Determine  the  appropriate  time  to  carry  out  the  assessment  taking  into  account,  normal 
shipboard  routine,  weather  /  sea  conditions,  and  status  of  the  equipment. 

4.  Review  the  working  space  in  and  around  the  lifeboat  to  determine  the  position  of  the 
assessor.  Assessor’s  position  should  allow  adequate  observation  of  the  individual’s 
performance. 

5.  Review  steps  necessary  to  ensure  safety. 

6.  Ensure  the  person  being  assessed  has  sufficient  knowledge,  training  and  experience  to 
complete  Objective  1 . 

Example  assessment  conditions 
Assessment  Objective 

Lifeboat  launching  pre-start  checks. 

Assessment  method 

Shipboard. 

Candidate  Orientation 

The  assessor  should  brief  the  candidate  on  what  objective  will  be  assessed. 

The  candidate  will  be  instructed  that  he/she  may  order  other  crewmembers  to  carry  out 
inspections.  He/she  must  use  proper  commands  and  supervise  the  crewmembers  closely, 
ensuring  the  inspection  is  carried  out  properly. 

Equipment,  Apparatus,  and  Tools 

Fully  functional  lifeboat  and  launching  equipment 

Initial  Equipment  Settings  or  Scenarios 

The  assessment  will  be  conducted  during  the  weekly  abandon  ship  drill  under  weather  and  sea 
conditions  that  will  allow  assessment  to  be  carried  out  safely. 

The  assessment  will  start  with  the  lifeboat  secured  in  the  stowed  position. 

Written  or  Oral  Questions 

None  provided. 
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EXERCISE  5 

Developing  Performance  Measures 


Time: 

Purpose: 

Outcome: 

Report: 


45  minutes 

Practice  developing  performance  measures 

List  of  multiple  performance  measures  for  selected  lifeboat  assessment  objectives 
List  the  measures  that  your  group  selected  (5  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  develop  performance  measures  for  up  to  6  lifeboat  launching 
assessment  objectives. 

Your  tools  for  this  exercise  are: 

□  Your  group’s  responses  to  Exercises  2-4  (assessment  objectives,  method,  and 
conditions). 

□  Attachment  1 ,  Developing  Performance  Measures  for  Rules  of  the  Road. 

□  Attachment  2,  Developing  Performance  Measures  for  Lookout  Procedures. 

□  Attachment  3,  Lifeboat  Launching  Performance  Measures  Worksheet. 

B.  To  get  started: 

□  Review  your  outputs  from  Exercises  2-4  (assessment  objectives,  method,  and 
conditions). 

□  Consider  the  basic  performance  requirements  of  lifeboat  launching. 

□  Define  specific  performance  measures  for  each  lifeboat  launching  assessment 
objective. 

□  Remember  that  performance  measures  are  procedures  for  observing  and  recording 
performance.  They  describe  either  observable  actions,  or  the  outcomes  of  those 
actions;  they  do  not  describe  how  well  the  candidate  has  to  perform  the  action(s). 

□  Try  to  include  both  process  and  product  measures. 

□  Please  do  NOT  specify  performance  standards',  they  will  be  the  focus  of  the  next 
exercise. 

For  examples  of  performance  measures  for  other  types  of  assessments,  review  Attachments  1  and 

2,  which  describe  Rules  of  the  Road  and  Lookout  Procedures  performance  measures. 

Record  your  group’s  answers  on  your  overhead  transparency  worksheet. 


C.  Select  a  spokesperson  to  tell  the  group  about  the  performance  measures  you  developed  and 
your  rationale. 
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EXERCISE  5 
Attachment  1 

Developing  Performance  Measures  for  Rules  of  the  Road2 

Introduction 

The  steps  involved  in  developing  performance  measures  and  standards  are  depicted  in  Figure  1 
below.  Since  Exercise  5  is  concerned  only  with  performance  measures,  only  Steps  1  and  2  are 
discussed  in  this  document.  Steps  3-6,  which  concern  performance  standards,  are  discussed  in 
Exercise  6,  Attachment  1 . 


Figure  1.  Summary  of  steps  in  the  development  of  performance  measures  and  scoring 
criteria. 

Steps  1  and  2  are  described  below  using  an  example  in  which  a  simulator  was  used  to  assess 
knowledge  of,  and  ability  to  apply,  Rules  of  the  Road  (ROR). 

Steps  in  the  Performance  Measure  Development  Process 

Step  1:  Match  the  performance  measures  to  the  assessment  objectives.  The  assessment 
objectives  were  considered,  and  three  basic  requirements  of  navigation  law  during  watch  were 

2  From  Smith,  M.W.,  Sandberg,  G.  R.,  McCallum,  M.C.,  Stewart,  R.,  Hard,  C.,  Meum,  R.,  and  Baker,  D.  (1996). 
An  interactive  test  of  mariner  competence .  U.S.  Coast  Guard  Report  No.  CG-D-04-97. 
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defined:  (1)  maintain  a  good  lookout  and  determine  if  risk  of  collision  exists;  (2)  take 
appropriate  action  or  maneuver  to  avoid  collision;  and  (3)  determine  if  own  ship’s  action  or 
maneuver  was  adequate  to  avoid  collision,  and  ensure  that  the  action  or  maneuver  does  not  put 
own  ship  in  a  close  quarters  situation  with  other  vessels. 

Step  2:  Refine  each  performance  measure.  Next,  the  general  performance  requirements 
defined  in  Step  1  were  extrapolated  into  a  series  of  specific  bridge  operation  measures.  Proposed 
bridge  operation  measures  addressed  visual  search,  binocular  viewing,  visual  bearing,  radar 
viewing,  use  of  sound  signals,  closest  point  of  approach  (CPA),  vessel  speed,  maneuver 
direction,  and  action  or  maneuver  to  avoid  collision.  Table  1  summarizes  the  bridge 
performance  measures  developed  in  this  example. 


Table  1.  Summary  of  bridge  performance  measures 


Performance  Measure 

Description 

Visual  Search 

Score  based  on  expert  mariners’  expected  time  spent 
visually  searching  each  of  four  quadrants  of  view  during 
the  scenario  observation  period. 

Binocular  Viewing 

Score  based  on  expert  mariners’  expected  frequency  of 
viewing  each  vessel  through  the  binoculars  during  the 
scenario  observation  period. 

Visual  Bearing 

Score  based  on  expert  mariners’  expected  frequency  of 
visual  bearings  of  each  vessel  to  be  taken  during  the 
scenario  observation  period. 

Radar  Viewing 

Score  based  on  expert  mariners’  expected  frequency  and 
total  duration  of  radar  viewings  during  the  scenario 
observation  period. 

Maneuver  Signal 

Proportion  of  times  a  correct  signal  was  sounded  within  30 
seconds  of  a  maneuver. 

CPA 

Score  based  on  expert  mariners’  expected  minimum  closest 
point  of  approach  to  other  vessels  throughout  each  test 
scenario. 

Radar  Maneuver 

Score  based  on  expert  mariners’  expected  frequency  and 
total  duration  of  radar  viewings  during  the  first  six  minutes 
after  the  first  own  ship  maneuver. 

Fog  Signal 

Score  based  on  expert  mariners’  expected  time  of  first 
sounding  of  the  restricted  visibility  sound  signal  during  the 
fog  scenario. 

Fog  Speed 

Score  based  on  expert  mariners’  expected  time  and  level  of 
action  to  reduce  speed  following  radar  failure  during  the 
fog  scenario. 

Action  or  Maneuver 

Action  or  maneuver,  and  projected  CPA  with  threat  vessel 
following  completion  of  maneuver  (scores  not  obtained  for 
analysis). 
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EXERCISE  5 
Attachment  2 

Developing  Performance  Measures 
for  Lookout  Procedures 

Discussion  of  requirements  and  constraints  for  Assessment  Objective  4 

1 .  Assume  shipboard  assessment  and/or  ship  bridge  simulator  assessment,  oral  or 
written  test  questions. 

2.  Review  the  preliminary  standards  under  Assessment  Objective  4  and  define 
performance  measures  for  each. 

Assessment  Objective  Measured 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  during  daylight. 

Preliminary  Standards:  Mariner  demonstrates  proper  consideration  and  correct  performance  of 
the  following  actions: 

1 .  Identifies  six  of  the  relevant  and  appropriate  sightings  that  should  be  reported  when 
detected  by  the  lookout. 

2.  Reports  sighted  surface  objects  using  ship’s  bell. 

3.  Reports  sighted  surface  objects  verbally  using  a)  degrees  and  b)  points. 

4.  Reports  sighted  sky  objects  verbally  using  points. 

5 .  Reports  audible  targets  verbally  using  points. 

Example  performance  measures  for  Assessment  Objective  4 

1 .  Candidate  answers  question  Identify  six  of  the  sightings  that  should  be  reported  when 
detected  by  the  lookout. 

2.  Candidate  reports  sighted  surface  objects  using  the  ship’s  bell. 

3.  Candidate  reports  sighted  surface  objects  verbally  using  degrees. 

4.  Candidate  reports  sighted  surface  objects  verbally  using  points. 

5.  Candidate  reports  sighted  sky  objects  verbally  using  points. 

6.  Candidate  reports  audible  targets  verbally  using  points. 
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EXERCISE  5 
Attachment  3 

Lifeboat  Launching  Assessment 
Performance  Measures  Worksheet 


ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 

ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 


ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 
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ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 


ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 


ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 
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EXERCISE  5 

Developing  Performance  Measures 
Example  Solution 


The  following  performance  measures  could  be  used  for  Assessment  Objective  1,  “person  in 
charge  carries  out  proper  inspections”: 

1 .  Candidate  inspects  the  releasing  gear. 

2.  Candidate  checks  releasing  gear  lever. 

3.  Candidate  inspects  tricing  pendants. 

4.  Candidate  inspects  sea  painter. 

5.  Candidate  checks  drain  plug. 

6.  Candidate  inspects  the  general  condition  and  arrangement  of  lifeboat  equipment. 

7.  Candidate  inspects  man  ropes. 

8.  Candidate  inspects  davit  tracks. 

9.  Candidate  inspects  trapping  lines. 

1 0.  Candidate  locates  hand  crank. 

1 1 .  Candidate  communicates  with  other  crewmembers  clearly  using  proper  terminology. 


NOTE:  The  assessor  observes  the  candidate’s  performance  of  the  above  tasks.  After  performing 
each  task,  the  candidate  reports  to  the  assessor  on  each  measure.  If  Assessment  Objective  2  is 
also  being  assessed,  then  the  candidate  will  be  expected  to  correct  any  discrepancies  found 
during  the  assessment  of  Objective  1. 
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EXERCISE  6 

Developing  Performance  Standards 


Time: 

Purpose: 

Outcome: 


Report: 


45  minutes 

Practice  developing  and  gaining  agreement  on  performance  standards 
Agreed  upon  standards  for  one  lifeboat  launching  assessment  objective  and  its 
associated  performance  measures 

Tell  group  what  standards  you  agreed  upon  and  why  (10  minutes) 


Instructions: 

A.  The  goal  of  this  exercise  is  to  develop  performance  standards  for  up  to  3  lifeboat  launching 
assessment  objectives.  Pretend  you  are  a  group  of  subject  matter  experts  convening  to 
discuss  the  performance  standards  and  scoring  criteria  for  a  lifeboat  launching  assessment. 

Your  tools  for  this  exercise  are: 

□  Your  group’s  responses  to  Exercise  5  (lifeboat  launching  performance  measures). 

□  Attachment  1 ,  Developing  Performance  Standards  for  Rules  of  the  Road. 

□  Attachment  2,  Developing  Performance  Standards  for  Lookout  Procedures. 

□  Attachment  3,  Lifeboat  Launching  Assessment  Performance  Standards  Worksheet. 

B.  To  get  started,  consider  the  performance  measures  you  developed  for  each  assessment 
objective  in  Exercise  5.  For  each  measure,  specify: 

□  The  standard  scoring  strategy  -  remember  that  the  strategy  can  be  either  pass/fail  or 
graded  (e.g.,  expert,  qualified,  not  qualified). 

□  Who  you  would  include  in  the  consensus-gathering  meeting. 

□  The  specifically  defined  performance  standards. 

□  Why  you  chose  these  standards. 

□  How  you  would  combine  measures  for  a  single  objective  (if  applicable). 

C.  For  examples  of  performance  standards  for  other  types  of  assessments,  review  Attachments  1 
and  2. 

Record  your  group’s  answers  on  the  overhead  transparency  worksheet. 

D.  Select  a  spokesperson  to  tell  the  larger  group  the  performance  standards  you  chose  and  why. 
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EXERCISE  6 
Attachment  1 

Developing  Performance  Standards  for  Rules  of  the  Road 


Introduction 

The  steps  involved  in  developing  performance  measures  and  standards  are  depicted  in  Figure  1 
below.  Since  this  exercise  is  concerned  with  performance  standards,  only  Steps  3-6  are  discussed 
in  this  document.  Steps  1  and  2  are  discussed  in  Exercise  5,  Attachment  1. 


Step  4:  Define  general  scoring 
strategy 


Step  5:  Get  SME  input  on  scoring 
criteria  for  each  measure 


Step  6:  Reach  group  consensus  on 
scoring  criteria 


Figure  1.  Summary  of  steps  in  the  development  of  performance  measures  and  scoring 
criteria. 

Steps  3-6  are  described  below  using  an  example  in  which  a  simulator  was  used  to  assess 
knowledge  of,  and  ability  to  apply,  Rules  of  the  Road  (ROR). 


3  From  Smith,  M.W.,  Sandberg,  G.  R.,  McCallum,  M.C.,  Stewart,  R.,  Hard,  C.,  Meum,  R.,  and  Baker,  D.  (1996). 
An  interactive  test  of  mariner  competence.  U.S.  Coast  Guard  Report  No.  CG-D-04-97. 
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Steps  3-6  in  the  Performance  Standard  Development  Process 

Step  3:  Define  basis  for  scoring.  Given  the  specific  performance  measures,  it  was  necessary  to 
define  the  basis  for  scoring.  In  discussions  among  the  expert  mariners,  it  was  determined  that 
one  of  two  general  approaches  toward  scoring  would  be  applied  to  each  bridge  operation 
measure.  In  most  cases,  a  Proficiency  Rating  was  applied.  In  selected  cases,  a  Proficiency 
Qualification  criterion  was  applied,  and  performance  was  scored  as  either  “competent”  or  “not 
competent.”  Levels  of  proficiency  and  competency  were  defined  as  follows: 

Proficiency  Rating  is  defined  with  reference  to  both  navigation  law  and  professional  standards. 
Considering  both,  it  is  the  consistency  of  performance  with  legally  mandated  actions,  as  defined 
by  navigation  law;  and  with  the  indicated  level  of  prudent  seamanship  (expert,  qualified, 
unqualified)  in  the  operational  application  of  navigation  law.  These  three  proficiency  ratings  are: 

□  Expert:  Performance  is  fully  consistent  with  all  legal  mandates  and  meets  the  highest 
professional  standards  of  prudent  seamanship  in  the  operational  application  of  navigation 
law. 

□  Qualified:  Performance  is  fully  consistent  with  all  legal  mandates  and  meets  acceptable 
professional  requirements  of  prudent  seamanship  in  the  operational  application  of 
navigational  law. 

□  Unqualified:  Does  not  meet  one  or  both  of  the  legally  mandated  actions  and/or 
acceptable  professional  requirements  of  prudent  seamanship  in  the  operational  application 
of  navigation  law. 

Proficiency  Qualification  is  defined  with  reference  to  navigation  law  only.  It  is  the  consistency 
of  performance  with  legally  mandated  actions,  as  defined  by  navigation  law.  The  two  levels  of 
competency  are: 

□  Qualified:  Performance  is  fully  consistent  with  legally  mandated  actions. 

□  Unqualified:  Performance  is  inconsistent  with  legally  mandated  actions. 

Step  4:  Define  general  scoring  strategy.  The  next  step  in  the  development  of  performance 
standards  called  for  the  consideration  of  the  general  scoring  strategy  to  be  employed  with  each 
measure.  Each  performance  measure  was  considered  in  turn,  and  the  basic  parameters  to  be 
considered  were  defined.  For  example,  in  the  case  of  Visual  Search,  it  was  determined  that  the 
Proficiency  Rating  approach  would  be  applied.  Additionally,  it  was  determined  that  one  set  of 
visual  search  measures  would  involve  the  percentage  of  a  test  participant’s  total  visual  search 
time  in  each  of  the  four  visual  quadrants  during  the  first  six  minutes  of  a  scenario.  Given  this 
definition,  it  was  then  recognized  that  ranges  of  percentages  would  be  required  to  define  the 
expert,  qualified  watchstander,  and  unqualified  levels  of  performance. 

Step  5:  Get  subject  matter  expert  (SME)  input  on  scoring  criteria  for  each  measure.  For 

each  bridge  operation  measure,  a  modified  Delphi  technique  was  employed  to  define 
performance  standards  corresponding  to  specific  Proficiency  Rating  or  Qualification.  Basically, 
the  Delphi  technique  involves  an  iterative  process  of  review,  independent  input,  and  group 
discussion  that  continues  until  a  group  consensus  is  reached.  During  this  process,  expert 
mariners  reviewed  the  operational  situation  for  a  given  test  scenario  then  individually 
recommended  performance  standards. 
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Step  6:  Reach  group  consensus  on  scoring  criteria.  The  set  of  recommended  performance 
standards  were  then  reviewed  by  the  group  of  expert  mariners  until  a  consensus  standard  was 
obtained  for  the  specific  measure  and  scenario.  Table  1  presents  the  results  of  this  process  for 
the  example  measure  of  Visual  Search  during  the  first  six  minutes  of  a  bridge  simulator  scenario. 
This  table  provides  the  specific  ranges  of  percentages  of  total  Visual  Search  time  that  the  team  of 
expert  mariners  judged  to  correspond  to  each  Proficiency  Rating  for  the  conditions  in  this 
scenario. 


Table  1.  Example  of  a  performance  standard  for  the  measure  of  visual  search 


Percentage  of  Total  Visual  Search  Time  in  Each  Direction  During 

First  Six  Minutes 

Performance 

Rating 

Forward 

Starboard 

Aft _ 

Port 

35-50 

30-45 

5-15 

5-15 

Expert 

25-34,  51-80 

15-29, 46-80 

16-35 

16-35 

Qualified 

Neither  Expert  nor  Qualified 

Unqualified 

Steps  5  and  6  of  this  process  were  repeated  for  each  bridge  operation  measure  in  each  scenario, 
resulting  in  quantitative  scoring  criteria  analogous  to  those  in  table  1  for  each  measure  in  each 
scenario. 
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EXERCISE  6 
Attachment  2 

Developing  Performance  Standards 
for  Lookout  Procedures 

Performance  standard  requirements  and  constraints: 

1 .  Assume  shipboard  assessment  and/or  ship  bridge  simulator  assessment,  oral  or  written 
test  questions. 

2.  Review  the  measures  under  Assessment  Objective  4  and  define  performance  standards 
for  each. 

Example  performance  standards  for  Assessment  Objective  4:  Demonstrate 
lookout  techniques  and  make  lookout  reports  in  clear  visibility  during  daylight. 


OBJECTIVE 

MEASURE 

STANDARD  1 

4. 1  List  the  sightings  that 

1  should  be  reported 

when  detected. 

Answers  question:  Identify 
six  of  the  sightings  that 
should  be  reported  when 
detected  by  the  lookout. 

Correct  response  includes  6  of  the 
following  sightings:  Distress 
signals;  persons  in  distress;  vessels 
and/or  aircraft  in  distress;  derelicts, 
wreckage,  floating,  or  partially 
submerged  debris;  pollution 
incident;  vessel  traffic  and/or 
aircraft;  aids  to  navigation  (nav. 
buoys  and  lights);  danger  to  the 
vessel  or  hazards  to  navigation; 
reduction  in  visibility  due  to  fog, 
mist,  falling  snow,  heavy 
rainstorms,  sandstorms,  or  any 
other  similar  cause;  change  in 
weather;  hearing  other  vessels  or 
aids  to  navigation;  ice;  unusual 
sightings  or  any  unreported  change 
in  the  field  of  view  or  sector; 
sightings  of  marine  life  in  f 

compliance  with  the  Endangered  f 

Species  Act  /  Marine  Mammal 
Protection  Act. 

4.2  Report  sighted 
objects  using  the 
ship’s  bell. 

Reports  sighted  objects  using 
the  ship’s  bell. 

Reports  at  least  3  surface  objects 
using  ship’s  bell.  Improper  ! 

reporting  constitutes  failure.  Failure 
to  detect  visible  objects  within  a  jj 

specified  sector  of  view  constitutes 
failure.  1 
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EXERCISE  6 
Attachment  2  (cont.) 


OBJECTIVE 

MEASURE 

STANDARD 

4.3  Report  sighted 

objects  verbally  using 
degrees. 

Reports  sighted  objects 
verbally  using  degrees. 

Verbally  reports  at  least  3  surface 
objects.  Reports  must  be  within 

221/2°  of  the  actual  bearing  of 
detected  targets. 

Improper  reporting  constitutes 
failure.  Failure  to  detect  a  visible 
object  within  a  specified  sector  of 
view  constitutes  failure. 

4.4  Report  sighted 

objects  verbally  using 
points. 

Reports  sighted  objects 
verbally  using  points. 

Verbally  reports  at  least  3  surface 
objects.  Reports  should  indicate: 

•  What  (type  of  objects) 

•  Where  (bearings,  relative  or 
true) 

•  How  far  off  (hull-down,  on 
the  horizon,  hull-up,  close 
aboard). 

Reports  must  be  within  +  2  points 
of  the  actual  bearing  of  the  detected 
targets. 

Improper  reporting  constitutes 
failure.  Failure  to  detect  a  visible 
object  within  a  specified  sector  of 
view  constitutes  failure. 

4.5  Report  sky  objects 
|  verbally  using  points. 

Reports  sighted  objects 
verbally  using  points. 

Verbally  reports  at  least  3  sky 
objects  using  points.  Reports  must 
be  within  +  2  points  of  the  actual 
bearing  of  detected  targets. 

Improper  reporting  constitutes 
failure.  Failure  to  detect  a  visible 
object  within  a  specified  sector  of 
view  constitutes  failure. 
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EXERCISE  6 
Attachment  2  (cont.) 


4.6  Report  audible  targets  Reports  audible  targets  Verbally  reports  at  least  3  audible 

verbally  using  points,  verbally  using  points.  targets.  Reports  should  indicate: 

•  What  (type  of  objects) 

•  Where  (bearings,  relative  or 
true) 

•  How  far  off  (hull-down,  on 
the  horizon,  hull-up,  close 
aboard). 

Reports  must  be  within  +  2  points 
of  the  actual  bearing  of  the  detected 
targets. 

Improper  reporting  constitutes 
failure.  Failure  to  detect  a  visible 
object  within  a  specified  sector  of 
_ view  constitutes  failure. 

FINAL  SCORE: 

□  Pass 

□  Fail 
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EXERCISE  6 
Attachment  4 

Lifeboat  Launching  Assessment 
Performance  Standards  Worksheet 


ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 

STANDARD  SCORING  STRATEGY  (pass/fail  or  graded): 

WHO  WOULD  BE  INCLUDED  IN  THE  CONSENSUS  GATHERING 
MEETING: 

PERFORMANCE  STANDARD(S): 

WHY  YOU  CHOSE  THESE  STANDARDS: 

HOW  YOU  WOULD  COMBINE  MEASURES  (if  applicable): 
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ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 

STANDARD  SCORING  STRATEGY  (pass/fail  or  graded): 

WHO  WOULD  BE  INCLUDED  IN  THE  CONSENSUS  GATHERING 
MEETING: 

PERFORMANCE  STANDARD(S): 

WHY  YOU  CHOSE  THESE  STANDARDS: 

HOW  YOU  WOULD  COMBINE  MEASURES  (if  applicable): 
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ASSESSMENT  OBJECTIVE: 

PERFORMANCE  MEASURE(S): 

STANDARD  SCORING  STRATEGY  (pass/fail  or  graded): 

WHO  WOULD  BE  INCLUDED  IN  THE  CONSENSUS  GATHERING 
MEETING: 

PERFORMANCE  STANDARD(S): 

WHY  YOU  CHOSE  THESE  STANDARDS: 

HOW  YOU  WOULD  COMBINE  MEASURES  (if  applicable): 
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EXERCISE  6 

Developing  Performance  Standards 
Example  Solution 


Performance  standard  requirements  and  constraints: 

□  Assume  shipboard  assessment  of  Assessment  Objective  1  “person  in  charge  carries  out 
proper  inspections.” 

□  Review  the  1 1  performance  measures  and  preliminary  standards  for  Assessment 
Objective  1  and  develop  a  set  of  performance  standards  for  each  performance  measure. 

□  Determine  the  basis  for  scoring  criteria.  Will  a  graded  or  pass/fail  strategy  be  used  for 
scoring? 

□  Determine  if  assessment  of  Objective  1  will  use  individual  or  combined  criteria.  Will  the 
failure  to  carry  out  one  or  more  of  the  performance  measures  properly  result  in  an 
automatic  failure  for  the  objective?  (Note:  In  an  overall  lifeboat  assessment,  it  must  be 
determined  if  the  failure  of  any  of  the  objectives  would  result  in  the  failure  of  the  lifeboat 
exam). 

Example  performance  standards  for  Assessment  Objective  1 : 

1.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

releasing  gear. 

Standard:  Candidate  correctly  reports  the  condition  of  the  releasing  gear. 

2.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

releasing  gear  lever. 

Standard:  Candidate  correctly  reports  the  condition  of  the  releasing  gear  lever. 

3.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

tricing  pendants. 

Standard:  Candidate  correctly  reports  the  condition  of  the  tricing  pendants. 

4.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of,  sea 

painter. 

Standard:  Candidate  correctly  reports  the  condition  of  the  sea  painter. 

5.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

drain  plug. 

Standard:  Candidate  correctly  reports  the  condition  of  the  drain  plug. 

6.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of,  the 

general  condition  and  arrangement  of  lifeboat  equipment. 

Standard:  Candidate  correctly  reports  the  condition  of  the  lifeboat  equipment. 
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7.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

man  ropes. 

Standard:  Candidate  correctly  reports  the  condition  of  the  man  ropes. 

8.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

davit  tracks. 

Standard:  Candidate  correctly  reports  the  condition  of  the  davit  tracks. 

9.  Measure:  Candidate  either  properly  inspects,  or  orders  and  supervises  the  inspection  of, 

trapping  lines. 

Standard:  Candidate  correctly  reports  condition  of  the  trapping  lines. 

10.  Measure:  Candidate  locates  the  hand  crank. 

Standard:  Candidate  correctly  reports  its  proper  location. 

11.  Measure:  Candidate  gives  orders  as  required. 

Standard:  Candidate  gives  clear,  distinct  orders  using  proper  terminology  without 
hesitation. 


NOTES: 

1 .  Assumes  graded  level  standard  strategy  is  used.  Suggested  levels  are: 

a)  Expert  -  Measure  is  accomplished  correctly  and  efficiently,  without  hesitation,  using 
proper  terminology. 

b)  Qualified  -  Measure  is  accomplished  correctly  with  some  thought  and  hesitation.  Minor 
mistakes  in  terminology  are  made  which  would  not  result  in  failure  to  launch  the  lifeboat. 

c)  Unqualified  -  Measure  is  not  accomplished,  or  accomplished  incorrectly.  Terminology 
used  is  incorrect  and  might  result  in  misunderstandings. 

2.  Failure  to  accomplish  measures  #lor  #2  at  the  qualified  level  would  result  in  the  automatic 
failure  of  the  objective.  If  the  releasing  gear  was  not  properly  engaged,  then  loss  of  the 
lifeboat  and/or  life  would  result. 
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A  METHOD  FOR  DEVELOPING  MARINER  ASSESSMENTS 


Chapter  3 


A  MANUAL  FOR  ASSESSMENT  DEVELOPERS 


This  chapter  describes  a  systematic,  step-by-step  method  for 
developing  Performance-based  Assessment  (PBA)  of  mariner 
proficiencies. 


This  report  may  be  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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INTRODUCTION 


Meeting  the  requirements  of  the  Seafarers  ’  Training,  Certification  and  Watchkeeping  Code 
(STCW  Code)  is  a  current  challenge  of  the  United  States  maritime  industry.  The  STCW  Code 
directs  the  United  States  maritime  industry  to  develop  procedures  for  assessing  mariner 
proficiency  on  the  basis  of  practical  demonstration.  The  STCW  Code  defines  areas  for  which 
mariners  must  receive  certification  and  specifies  the  need  to  assess  proficiency  on  the  basis  of 
skill  and  knowledge  demonstrations  by  the  mariner.  However,  there  has  been  an  ongoing  debate 
regarding  what,  precisely,  is  implied  by  the  STCW  Code  directive  for  assessments  that  address: 

. . .  the  proper  performance  of  functions  onboard  ship  in  accordance  with  the 
internationally  agreed  criteria  as  set  forth  herein  and  incorporating  prescribed  standards 
or  levels  of  knowledge,  understanding,  and  demonstrated  skill . . .  (STCW  Code,  Section 
A- 1/1,  paragraph  1.1). 

This  chapter  is  intended  to  be  used  as  a  manual  and  be  considered  as  such  and  is  referred  to  as 
such  throughout  this  chapter.  The  intent  of  this  manual  is  to  outline  a  process  that  can  be  used  by 
professional  mariners  in  training  institutions  and  other  organizations  to  develop  mariner 
assessment  procedures.  The  process  described  in  this  manual  conforms  to  international 
standards  and  domestic  regulations,  especially  the  International  Maritime  Organization’s  STCW 
Code  and  the  U.S.  Coast  Guard’s  Navigation  and  Vessel  Inspection  Circulars  (NVICs)  that 
address  implementation  of  the  STCW  Code  within  the  United  States.  The  reference  section  of 
this  manual  lists  specific  STCW  documents  and  applicable  NVICs. 

This  manual  is  based  on  a  comprehensive  methodology  for  mariner  assessment  development 
documented  in  the  first  chapter  of  this  report.  The  methodology  is  also  presented  in  a  set  of 
workshop  materials  and  exercises  entitled  A  Workshop  for  Assessment  Developers  (Chapter  2). 
More  generally,  this  manual  relies  heavily  on  the  general  method  of  Instructional  Systems 
Development  (ISD).  More  detailed  reference  information  on  all  of  the  above  sources  is  provided 
in  the  reference  section  of  this  manual. 

The  objective  behind  the  process  outlined  in  this  manual  is  to  develop  valid  and  reliable  mariner 
assessment  procedures.  Valid  assessments  evaluate  mariners  on  the  basis  of  their  demonstration 
of  job-critical  knowledge,  skills,  and  abilities.  Valid  assessments  serve  as  accurate  predictors  of 
a  mariner’s  abilities  to  fulfill  the  functional  requirements  of  a  job  or  task.  Reliable  assessments 
consistently  obtain  the  same  assessment  results  when  different  mariners  with  comparable  skills 
are  assessed.  An  assessment  must  be  reliable  and  evaluate  job-critical  requirements  in  order  to 
be  valid. 

The  format  of  an  assessment  procedure  is  not  essential  to  the  validity  and  reliability  of  that 
procedure.  Differences  in  organizational  goals,  assessment  objectives,  and  performance 
standards  preclude  prescribing  a  single  format  for  assessments.  However,  a  sample  assessment 
that  proved  to  be  useful  during  the  development  of  this  process  is  provided  in  this  manual  as  an 
aid  to  the  reader. 


The  remainder  of  this  manual  consists  of  a  set  of  instructions  for  developing  mariner  assessment 
procedures,  a  glossary,  a  reference  section,  and  a  sample  assessment  procedure  that  is  referred  to 
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throughout  the  instructions.  The  instructions  are  intended  to  provide  a  relatively  straightforward 
presentation  of  the  steps  needed  to  develop  a  valid  and  reliable  assessment  procedure.  Issues 
critical  to  the  validity  and  reliability  of  an  assessment  procedure  are  discussed,  as  necessary.  The 
glossary  provides  a  set  of  terms  and  definitions  that  are  internally  consistent  with  one  another 
and,  to  the  extent  possible,  consistent  with  current  STCW,  U.S.  Code  of  Federal  Regulations 
(CFR),  and  NVIC  terms  and  definitions.  The  reference  section  identifies  the  international  and 
domestic  regulatory  source  documents;  other  manuals,  authoritative  books,  and  articles  that 
address  assessment  development;  and  selected  recent  research  on  mariner  assessment  sponsored 
by  the  U.S.  Coast  Guard  (USCG). 

Instructions 

This  manual  identifies  a  series  of  steps  that  can  be  conducted  to  develop  a  valid  and  reliable 
assessment  procedure  that  meets  International  Maritime  Organization  (IMO)  and  USCG 
requirements  for  mariner  demonstration  of  proficiency  in  selected  areas.  Following  is  an  outline 
of  the  five  basic  steps  and  associated  sub-steps  described  in  this  manual. 

1 .0  Specify  Assessment  Objectives. 

1 . 1  Identify  regulatory  requirements. 

1 .2  Analyze  job  requirements. 

1.3  Describe  individual  assessment  objectives. 

2.0  Determine  Assessment  Methods. 

2. 1  Identify  alternative  assessment  methods. 

2.2  Review  the  advantages  and  disadvantages  of  alternative  assessment  methods. 

2.3  Determine  assessment  methods. 

3.0  Specify  Assessment  Conditions. 

3.1  Describe  assessment  settings  or  scenarios. 

3.2  Specify  oral  and  written  questions. 

4.0  Develop  Proficiency  Criteria. 

4.1  Identify  components  of  assessment  objectives. 

4.2  Develop  individual  measures  and  standards. 

4.3  Develop  proficiency  criteria  and  scoring  procedures. 

4.4  Validate  measures,  standards,  proficiency  criteria,  and  scoring  procedures. 
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5.0  Prepare  the  Assessment  Materials. 

5. 1  Prepare  detailed  assessment  worksheets. 

5.2  Prepare  assessor  instructions. 

5.3  Prepare  candidate  instructions 

5.4  Finalize  oral  and  written  questions. 

5.5  Finalize  simulator  and/or  shipboard  procedures. 

5.6  Document  references,  source  materials,  and  the  validation  process. 

The  following  instructions  are  presented  in  a  standard  format  to  facilitate  review  and  reference. 
Following  the  title  of  each  step,  the  goal  of  the  step  is  briefly  described,  followed  by  a  listing  of 
the  sub-steps  for  that  step.  Then,  separate  instructions,  discussions,  and  examples  are  presented 
to  address  the  associated  procedures,  issues,  and  output. 

1 .0  Specify  Assessment  Objectives 

Goal  of  this  step 

Identify  the  knowledge,  skills,  and  abilities  on  which  the  mariner  will  be  assessed. 

Sub-steps 

1 . 1  Identify  regulatory  requirements. 

1.2  Analyze  job  requirements. 

1.3  Describe  individual  assessment  objectives. 

1.1  Identify  regulatory  requirements 

A  mariner  assessment  procedure  should  be  explicitly  associated  with  a  specific  requirement  in 
the  STCW  Code.  The  format  of  the  STCW  Code  provides  five  levels  of  requirements  that  are 
readily  identifiable,  as  shown  in  the  following  example: 

Chapter:  II  -  Standards  regarding  the  master  and  deck  department. 

Table:  A-II/1  -  Specification  of  minimum  standard  of  competence  for  officers  in  charge  of 
a  navigational  watch  on  ships  of  500  gross  tonnage  or  more. 

Function:  Navigation  at  the  operational  level. 

Competence:  Plan  and  conduct  a  passage  and  determine  position. 

Knowledge,  understanding  and  proficiency:  Celestial  navigation  —  Ability  to  use  celestial 
bodies  to  determine  the  ship’s  position. 

To  specify  the  assessment  objectives  for  an  assessment  procedure,  begin  by  identifying  all  of  the 
STCW  Code  requirements  corresponding  to  these  five  levels.  Later,  this  information  will  be 
included  in  the  assessment  documentation.  In  the  Lookout  assessment  example  in  Appendix  A, 
the  STCW  Reference  Information  entries  in  the  Assessment  Control  Sheet  provide  similar 
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information  from  the  STCW  Code. 


In  developing  an  assessment  to  meet  certification  or  licensing,  other  regulatory  requirements 
should  also  be  addressed.  For  example,  the  CFR  provides  explicit  guidance  regarding 
lifeboatman  examination  and  demonstration  of  ability.  If  the  assessment  is  intended  to  satisfy 
these  additional  regulatory  requirements,  the  requirements  should  be  explicitly  addressed  by  the 
assessment  and  referred  to  in  the  assessment  documentation. 

1 .2  Analyze  job  requirements 

STCW  Code  and  regulatory  requirements  typically  will  not  provide  sufficiently  detailed 
information  to  support  the  development  of  a  mariner  assessment.  To  fill  this  gap,  assessment 
developers  should  refer  to  technical  manuals,  job  instructions,  textbooks,  and  task  analyses  to  aid 
in  the  identification  of  job  requirements.  If  such  sources  are  used  to  derive  assessment 
objectives,  they  should  be  identified  in  the  assessment  documentation.  For  the  present  Lookout 
assessment  example,  the  STCW  Code  specifies  the  following  knowledge,  understanding,  and 
proficiency:  “ Responsibilities  of  a  lookout,  including  reporting  the  appropriate  bearing  of  a 
sound  signal,  light,  or  other  object  in  degrees  or  points .”  This  information  from  the  STCW  Code 
was  used  to  define  the  general  requirements,  which  were  augmented  by  a  review  of  46  CFR, 
Chapter  1,  Subpart  97.27  -  Lookouts,  and  by  the  Navigation  Rules  (Rule  2  -  Responsibility  and 
Rule  5  -  Lookout).  Rule  5  states:  “ Every  vessel  shall  at  all  times  maintain  a  proper  look-out  by 
sight  and  hearing  as  well  as  by  all  available  means  appropriate  in  the  prevailing  circumstances 
and  conditions  so  as  to  make  a  full  appraisal  of  the  situation  and  of  the  risk  of  collision.  ” 

In  most  cases,  an  individual  assessment  area  will  be  analyzed  to  identify  a  set  of  several  related 
assessment  objectives  that  will  be  addressed  separately  during  a  proficiency  assessment.  Each  of 
these  objectives  should  represent  a  critical  component  of  the  job  requirement  being  analyzed.  At 
this  point,  the  goal  is  to  identify  those  broad  areas  of  knowledge,  skill,  and  ability  that  are  critical 
to  successful  job  performance.  Later,  during  Step  4.0,  a  more  detailed  task  analysis  will  be 
conducted.  There  are  numerous  job  analysis  methods  that  might  be  used  in  defining  assessment 
objectives.  Further  guidance  on  the  conduct  of  job  and  task  analysis  can  be  found  in  selected 
references  listed  in  this  manual  (e.g.,  Jonassen,  Hannum,  &  Tessmer,  1989;  Mager,  1997; 
Rothwell  &  Kazanas,  1998).  Review  of  available  texts,  along  with  review  and  discussion  from 
qualified  professionals,  should  result  in  an  adequate  set  of  assessment  objectives. 

An  individual  assessment  procedure  must  meet  the  stated  STCW  Code  requirements  and  be 
appropriate  in  scope  in  order  to  be  practically  feasible.  An  assessment  should  address  a  single 
job  requirement  or  integrated  set  of  job  requirements  defined  in  the  STCW  Code.  In  addition, 
the  time  and  resource  requirements  for  assessment  should  be  within  a  practical  level.  An 
assessment  procedure  is  too  broad  in  scope  if  it  addresses  a  wide  range  of  unrelated  job 
requirements  defined  in  the  STCW  Code  and/or  requires  an  unnecessarily  extensive  amount  of 
time  and  resources. 

1 .3  Describe  individual  assessment  objectives 

Each  assessment  objective  should  include  a  general  description  of  the  mariner  performance  that 
will  be  required  during  assessment  and  the  conditions  under  which  they  are  to  be  performed. 

The  performance  should  be  described  as  a  visible,  overt  action  to  be  demonstrated  to  the 
assessor.  At  this  point,  conditions  should  identify  any  special  characteristics  of  the  assessment 
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setting,  such  as  day  or  night  conditions  and  major  types  of  equipment  or  equipment  settings 
required  for  the  proficiency  assessment.  Table  1  lists  the  nine  assessment  objectives  specified 
for  the  Lookout  assessment.  This  example  illustrates  very  broad  condition  descriptions,  limited 
to  day,  night,  and  restricted  visibility  conditions. 


Table  1.  Summary  of  Lookout  assessment  objectives. 


1. 

Describe  lookout  duties  and  responsibilities. 

2. 

Identify  lookout  stations  and  safe  routes  onboard. 

3. 

Describe  and  identify  the  international  distress  signals. 

4. 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  during  daylight. 

5. 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  at  night. 

6. 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  restricted  visibility. 

7. 

Demonstrate  the  use  of  lookout  equipment. 

8. 

Demonstrate  man  overboard  procedures. 

9. 

Demonstrate  lookout  watch  relief  procedures. 

2.0  Determine  Assessment  Methods 

Goal  of  this  step 

Select  the  most  appropriate  method  for  conducting  the  assessment  of  each  objective. 

Sub-steps 

2. 1  Identify  alternative  assessment  methods. 

2.2  Review  the  advantages  and  disadvantages  of  alternative  assessment  methods. 

2.3  Determine  assessment  methods. 

2.1  Identify  alternative  assessment  methods 

The  intent  of  the  STCW  Code  is  to  foster  assessment  through  practical  demonstration.  Thus,  it  is 
important  that  assessment  developers  carefully  consider  the  alternative  assessment  methods 
available  to  them  and  attempt  to  select  the  method  that  best  meets  the  intent  of  the  STCW  Code, 
while  also  considering  the  practical  limitations  of  mariner  assessment. 

This  step  begins  with  the  consideration  of  alternative  assessment  methods  for  each  assessment 
objective  identified  in  the  preceding  step.  There  are  three  basic  methods  of  assessment: 

•  Oral  or  written  questions. 

•  Exercises  in  a  simulated  job  setting. 

•  Exercises  in  an  actual  shipboard  job  setting. 

Oral  or  written  questions  ask  the  candidate  to  respond  to  factual  questions  regarding  general  or 


3-9 


A  Method  for  Developing  Mariner  Assessments 


Chapter  3- A  Manual  for  Assessment  Developers 


job-specific  knowledge.  Simulated  job  settings  vary  widely  in  the  extent  to  which  they 
accurately  present  cues  from  work  settings  and  require  candidates  to  respond  in  the  same  manner 
as  onboard  ship.  Assessments  conducted  in  the  actual  shipboard  job  setting  provide  the 
candidate  with  actual  work  setting  cues  and  require  candidate  responses  that  reflect  actual  job 
requirements.  The  reader  should  note,  however,  that  even  in  an  actual  shipboard  job  setting,  the 
conditions  of  assessment  frequently  are  modified  from  non-assessment  conditions.  For  example, 
at  the  onset  of  assessment  it  might  be  necessary  to  provide  the  candidate  with  a  more  explicit 
explanation  of  the  current  situation  than  is  typically  provided  during  normal  operations. 

At  the  most  general  level,  there  are  two  basic  categories  of  what  is  being  assessed  by  an 
assessment  procedure,  as  summarized  in  Table  2.  First,  an  assessment  can  evaluate  the 
candidate’s  demonstration  of  knowledge  of  concepts,  constructs,  rules,  and  procedures.  Often, 
specific  areas  of  knowledge  can  be  identified  as  a  prerequisite  to  proficiency  in  job  performance. 
In  such  cases,  assessment  of  a  candidate’s  knowledge  is  appropriate.  Second,  an  assessment  can 
evaluate  the  candidate’s  demonstration  of  skills  and  ability  to  apply  knowledge  of  concepts, 
constructs,  rules,  and  procedures  in  an  operational  setting.  Demonstration  of  ability  in 
operational  settings  provides  a  much  more  valid  basis  of  assessment.  However,  sometimes  the 
assessment  of  knowledge  demonstration  in  a  non-operational  setting  is  more  appropriate,  due  to 
economic,  efficiency,  or  safety  concerns. 

The  general  category  of  what  is  being  assessed  corresponds  to  the  general  method  of  mariner 
assessment,  as  summarized  in  the  right-hand  column  of  Table  2.  As  presented  here,  oral  or 
written  questions  are  used  to  assess  the  mariner’s  demonstration  of  knowledge.  Simulated  or 
actual  shipboard  job  settings  are  used  to  assess  the  mariner’s  demonstration  of  the  ability  to 
apply  knowledge  in  an  operational  setting. 


Table  2.  General  guidance  on  the  selection  of  assessment  methods. 


What  is  Being  Assessed 


Method  of  Mariner  Assessment 


Knowledge  of  concepts,  constructs,  rules,  and 
procedures 

Skills  and  ability  to  apply  knowledge  in  an 
operational  setting 


Oral  or  written  questions 


Practical  demonstration  in  real  or 
simulated  operations 


2.2  Review  the  advantages  and  disadvantages  of  alternative  assessment  methods 

Once  the  alternative  assessment  methods  have  been  identified,  the  assessment  developer  should 
consider  the  advantages  and  disadvantages  of  each  alternative.  Factors  to  consider  are 
assessment  validity,  assessment  reliability,  ease  of  assessment  development,  and  ease  of 
assessment  administration.  Recall  that  a  valid  assessment  is  one  that  represents  critical 
performance  in  the  operational  setting.  A  reliable  assessment  is  one  that  produces  consistent 
results  from  assessor  to  assessor,  and  candidate  to  candidate.  Table  3  summarizes  some  of  the 
common  advantages  and  disadvantages  associated  with  each  of  the  three  general  assessment 
methods. 
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Table  3.  Assessment  method  advantages  and  disadvantages. 


Method 

Advantage 

Disadvantage 

Oral  or  written 
questions 

.  Easy  to  standardize 

Easy  to  test  many  at  once 

Good  measure  of  knowledge 

May  not  adequately  represent  actual  requirements  of 
work  setting  (validity) 

Simulator 

demonstration 

High  reliability 

Safe 

May  have  high  validity 

May  not  adequately  represent  actual  requirements  of 
work  setting  (validity) 

May  be  expensive 

Usually  need  one  assessor  per  candidate 

Shipboard 

demonstration 

High  validity 

Usually  need  one  assessor  per  candidate 

More  difficult  to  control  conditions  (reliability) 

Task  may  be  infrequent  or  dangerous 

2.3  Determine  assessment  methods 

The  assessment  developer  should  consider  each  assessment  objective  and,  in  turn,  determine  an 
assessment  method.  The  specification  of  the  assessment  method  should  include  a  general 
description  of  the  assessment  method  and  assessment  conditions  for  each  objective.  Assessment 
methods  and  conditions  must  reflect  a  reasonable  level  of  operational  job  requirements.  The 
assessment  methods  must  also  provide  a  means  of  maintaining  sufficient  control  over  events  so 
that  mariner  safety  is  ensured  and  operational  risks  are  minimized. 

The  assessment  methods  defined  for  the  Lookout  assessment  were  initially  divided  between  (1) 
oral  or  written  questions  and  (2)  practical  demonstration  in  shipboard  or  simulated  operations. 
This  provided  flexibility  for  future  applications  in  the  use  of  either  oral  or  written  and  either 
simulator  or  shipboard  assessment.  Table  4  summarizes  the  assessment  method  selected  for  each 
Lookout  assessment  objective  from  the  sample  assessment  procedure. 
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Table  4.  Lookout  assessment  objectives  and  methods. 


Assessment  Objective 

Assessment  Method 

Describe  lookout  duties  and  responsibilities. 

Oral  or  written  questions 

Identify  lookout  stations  and  safe  routes  onboard. 

Oral  or  written  questions 

Describe  and  identify  the  international  distress 
signals. 

Oral  or  written  questions 

Demonstrate  lookout  techniques  and  make  lookout 
reports  in  clear  visibility  during  daylight. 

Practical  demonstration  -  shipboard  and/or  ship 
bridge  simulator 

Demonstrate  lookout  techniques  and  make  lookout 
reports  in  clear  visibility  at  night. 

Practical  demonstration  —  shipboard  and/or  ship 
bridge  simulator 

Demonstrate  lookout  techniques  and  make  lookout 
reports  in  restricted  visibility  during  daylight  or  at 
night. 

Practical  demonstration  —  shipboard  and/or  ship 
bridge  simulator 

Demonstrate  the  use  of  lookout  equipment. 

Practical  demonstration  — shipboard  and/or  ship 
bridge  simulator 

Demonstrate  man  overboard  procedures. 

Practical  demonstration  —  shipboard  and/or  ship 
bridge  simulator 

Demonstrate  lookout  watch  relief  procedures. 

Practical  demonstration  —  shipboard  and/or  ship 
bridge  simulator 

3.0  Specify  Assessment  Conditions 

Goal  of  this  step 

Outline  the  assessment  conditions  for  the  candidate  and  assessor. 

Sub-steps 

3 . 1  Describe  assessment  settings  or  scenarios. 

3.2  Specify  oral  and  written  questions. 

3.1  Describe  assessment  settings  or  scenarios 

Following  the  determination  of  assessment  objectives  and  methods,  the  assessment  developer 
should  begin  preparing  a  more  detailed  description  of  how  the  assessment  will  be  conducted. 
The  assessment  conditions  provide  a  good  point  of  departure  for  this  detailed  specification.  The 
following  should  be  included  in  a  detailed  description  of  the  assessment  conditions: 

•  Assessor  instructions. 

•  Candidate  instructions. 

•  Test  conditions,  tools,  and  apparatus  used  by  candidate. 
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•  Specific  simulator  scenario  requirements. 

•  Requirements  for  shipboard  operations. 

The  assessor  instructions  should  address  special  assessment  requirements  for  which  the  assessor 
needs  to  prepare  in  advance.  In  the  case  of  simulator-based  assessments,  this  includes 
familiarization  with  the  controls,  displays,  and  outputs  of  the  simulator.  In  the  case  of  shipboard 
assessments,  it  includes  specific  operational  conditions  required  to  ensure  a  valid  and  reliable 
assessment.  For  example,  if  a  lookout  assessment  is  being  conducted  onboard  a  vessel,  it  may  be 
necessary  to  have  a  specific  number  of  targets  visible  during  the  course  of  the  assessment.  In 
addition,  issues  of  personnel  safety,  operational  risk,  and  operational  efficiency  should  be 
identified  at  this  point. 

Candidate  instructions  should  provide  the  candidates  with  the  information  they  will  need  to 
prepare  for  assessment.  If  simulators  are  being  used,  a  detailed  description  of  the  operational 
scenario  should  be  provided,  specifying  both  the  initial  settings  of  equipment  and  how  the 
candidate  will  be  allowed  to  operate  controls  during  the  course  of  the  assessment.  For  shipboard 
assessment,  the  operations  under  which  assessment  is  to  be  conducted  must  be  described  and  the 
initial  settings  of  operational  equipment  should  also  be  specified.  This  description  requires 
comparable  detail  to  that  for  simulator  assessment,  although  the  ability  of  the  assessor  to  control 
these  conditions  may  be  more  limited.  Table  5  provides  a  summary  of  the  assessment  conditions 
described  for  Lookout  assessment  Objective  4. 


Table  5.  Summary  of  assessment  conditions  for  Lookout  assessment  Objective  4. 


Assessment  Objective 

Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility 
during  daylight. 

Initial  Condition 

The  assessment  should  be  conducted  in  clear  visibility  during  daylight.  It  is 
essential  that  reportable  objects  are  in  sight. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an 
internal  communications  system,  ship’s  bell,  7  X  50  individual  eye  focus 
binoculars,  and  bearing  repeater  fitted  with  a  bearing/azimuth  circle, 
alidade,  or  peloris.  The  lookout  station  should  be  clear  and  the  assessor 
must  be  able  to  observe  activities. 

3.2  Specify  oral  and  written  questions 

Oral  and  written  questions  should  be  adequately  specified  to  ensure  comprehensive  and 
consistent  assessment.  The  topics  to  be  addressed  by  the  set  of  questions  should  be  adequately 
defined  so  that  this  part  of  the  assessment  will  address  a  common  set  of  concepts,  constructs, 
rules,  or  problem-solving  skills  from  one  assessment  to  the  next.  In  the  best  case,  a  library  of 
test  items  will  be  developed,  along  with  procedures  for  sampling  from  this  library  for  any 
assessment.  At  a  minimum,  the  number  of  questions  should  be  identified,  as  well  as  the  topics 
and  subtopics  for  subsets  of  questions.  In  addition,  the  type  of  question  (e.g.,  open-ended, 
multiple-choice,  fill-in,  and  essay/discussion)  should  be  specified. 

Table  6  lists  the  oral  questions  included  in  the  Lookout  assessment,  providing  more  detail  on  the 
assessment  objectives  and  methods  identified  in  Table  4.  These  questions  address  assessment 


3-13 


A  Method  for  Developing  Mariner  Assessments 


Chapter  3- A  Manual  for  Assessment  Developers 


Objectives  1,  2,  and  3.  The  question  format  is  open-ended.  The  candidate  is  instructed  to 
provide  complete  answers,  and  the  assessor  is  asked  not  to  provide  coaching  or  feedback  during 
the  questioning  period. 


Table  6.  Oral  questions  from  the  Lookout  assessment. 


Assessment  Objective 

1 .  Describe  lookout  duties  and 
responsibilities. 

2.  Identify  lookout  stations  and  safe  routes 
onboard. 


3.  Describe  and  identify  the  international 
distress  signals. 


Written  or  Oral  Test  Questions 

What  are  the  duties  and  responsibilities  of  the 
lookout? 

Identify  three  common  lookout  stations  on  this  vessel. 

Identify  the  safe  routes  to  the  bow  lookout  station  on 
this  vessel. 

Describe  and  identify  six  international  distress 
signals. 


4.0  Develop  Proficiency  Criteria 

Goal  of  this  step 

Specify  the  criteria  that  will  be  used  to  assess  mariner  proficiency,  including  the  measures, 
standards,  proficiency  criteria,  and  scoring  procedures. 

Sub-steps 

4.1  Identify  components  of  assessment  objectives. 

4.2  Develop  individual  measures  and  standards. 

4.3  Develop  proficiency  criteria  and  scoring  procedures. 

4.4  Validate  measures,  standards,  proficiency  criteria,  and  scoring  procedures. 

4.1  Identify  components  of  assessment  objectives 

To  this  point,  assessment  objectives  have  been  defined  that  correspond  to  basic  job  requirements 
identified  in  the  STCW  Code,  other  regulatory  requirements,  or  job/task  descriptions.  The 
performance  required  by  an  objective  may  need  to  be  broken  into  smaller,  more  discrete 
components  for  assessment  purposes.  The  separate  components  of  an  objective  consist  of 
knowledge  and/or  application  of  concepts  or  constructs,  rules  of  operation,  or  procedures  for 
operation.  Separate  sources  of  information  are  available  to  assist  in  the  identification  of 
components  corresponding  to  each  of  these  three  types. 

Three  separate  information  sources  are  recommended  in  Table  7  for  each  type  of  assessment 
objective  component,  with  each  set  including  a  type  of  expert.  In  general,  it  is  recommended 
that  broadly  referenced  sources  (i.e.,  textbooks,  regulations,  and  technical  manuals)  be  reviewed 
first.  These  can  be  followed  by  more  specific  references  developed  for  particular  applications 
(i.e.,  course  materials,  company  policy,  and  ship  manuals).  Finally,  experts  (i.e.,  training 
experts,  regulation  and  policy  experts,  and  operational  experts)  should  be  consulted.  Information 
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that  is  more  specific  to  a  course,  company,  or  ship  is  often  required  to  identify  assessment 
components  that  realistically  reflect  job  requirements.  Frequently,  limited  reference  material 
will  be  available,  requiring  greater  reliance  on  expert  input.  It  is  critical  to  ensure  the  validity  of 
expert  information  because  there  are  many  perspectives  regarding  concepts,  interpretations  of 
rules,  and  operationally  specific  steps  and  procedures.  Assessment  should  address  commonly 
accepted  assessment  objective  components.  Therefore,  experts  should  be  selected  with  care,  and 
input  from  multiple  sources  should  be  obtained  and  compared. 

Table  7.  Recommended  information  sources  for  identifying  components  of 


assessment  objectives. 


Type  of  Assessment  Objective  Component 

Recommended  Information  Sources 

Concepts  or  constructs 

Textbooks 

Course  materials 

Training  experts 

Rules  of  Operation 

Regulations 

Organizational  policies 

Regulation  and  policy  experts 

Steps  or  Procedures  of  Operations 

Technical  manuals 

Ship  manuals 

Operations  experts 

For  the  Lookout  assessment,  three  components  of  assessment  Objective  8,  Demonstrate  man 
overboard  techniques,  were  identified  on  the  basis  of  a  review  of  ship  manuals  and  interviews 
with  operations  experts.  These  were  (1 .)  throw  a  ring  life  buoy,  (2.)  sound  the  alarm,  and  (3.) 
keep  the  victim  in  sight  at  all  times. 

4.2  Develop  individual  measures  and  standards 

Reliable  and  valid  assessment  requires  the  consistent  measurement  of  mariner  performance  and 
the  consistent  application  of  relevant  performance  standards.  This  process  can  sometimes  be 
relatively  complex.  As  a  consequence,  the  following  discussion  is  the  most  involved  in  this 
manual.  The  instructions  for  measure  and  standard  development  rely  upon  the  definition  of 
several  key  terms,  as  discussed  in  the  following  paragraphs. 

A  performance  measure  is  a  description  of  how  a  candidate’s  performance  is  to  be  observed  and 
recorded.  A  useful  distinction  between  types  of  measures  contrasts  instrumented  and 
observational  measures.  An  instrumented  measure  relies  upon  apparatus  that  can  be  calibrated 
to  yield  highly  consistent  and  accurate  measurement  results.  Examples  of  apparatus  used  in 
instrumented  measurement  are  stopwatches,  compasses,  rudder  angle  indicators,  oil  pressure 
gauges,  and  volt  meters.  An  example  of  an  instrumented  performance  measure  is  the  time 
required  for  launching  a  lifeboat.  A  complete  description  of  how  this  measure  is  to  be  applied 
includes  (1)  how  to  signal  the  start  of  the  launching  assessment,  (2)  when  to  start  the  timer,  (3) 
the  type  of  timer  used  and  its  calibration,  and  (4)  when  to  stop  the  timer  and  record  the  launch 
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time.  Whenever  practical  and  meaningful,  instrumented  measures  should  be  used  in  assessment 
because  they  provide  the  most  reliable  means  of  measurement. 

An  observational  measure  relies  primarily  on  the  assessor’s  observation  and  interpretation  of 
mariner  performance.  In  applying  an  observational  measure,  an  assessor  will  observe  and 
categorize  the  mariner’s  performance  with  reference  to  an  established  standard.  A  description  of 
how  an  observational  measure  is  to  be  applied  is  comprised  of  the  recording  form  and  the 
instructions  for  how  to  interpret  the  observed  performance.  Although  not  as  inherently 
consistent  and  accurate  as  instrumented  measures,  the  consistency  and  accuracy  of  observational 
measures  can  be  greatly  enhanced  by  carefully  describing  how  the  assessor  should  apply  the 
established  standards  to  the  observed  performance. 

A  performance  standard  is  the  level  of  a  measure  that  is  established  as  an  acceptable  or  target 
level.  In  the  case  of  an  instrumented  measure,  the  acceptable  level  is  established  independently 
of  the  observational  procedure.  For  the  example  above  describing  the  observation  of  the  time  to 
launch  a  lifeboat,  the  standard  would  be  an  established  time  limit  that  must  be  met  for  acceptable 
performance.  In  the  case  of  an  observational  measure,  the  standard  is  established  as  a  part  of  the 
observational  procedure.  If  the  assessor  interprets  the  observed  performance  as  matching  the 
important  characteristics  of  the  established  standard,  the  performance  is  acceptable. 

Figure  1  outlines  a  general  procedure  for  developing  measures  and  standards.  The  process  is 
linear,  with  the  exception  of  one  branch  that  must  be  taken  on  the  basis  of  the  type  of 
performance  measure  that  is  used.  The  following  discussion  describes  each  step  in  this  process. 


Figure  1.  General  process  for  developing  measures  and  standards. 

Characterize  the  mariner  action  or  outcome.  The  first  step  in  this  process  is  to  characterize 
the  actions,  or  outcomes  of  actions,  performed  by  the  mariner  into  one  of  three  categories.  First, 
the  mariner’s  action  may  be  a  verbal  or  written  response  to  a  question.  Second,  the  mariner’s 
action  may  be  the  performance  of  a  sequence  of  operational  actions.  Third,  the  mariner’s  action 
may  result  in  an  outcome  that  can  be  readily  observed  by  the  assessor.  The  characteristics  of 
each  mariner  action  or  outcome  should  be  described  to  allow  subsequent  consideration  of  the 
corresponding  performance  requirements.  Table  8  provides  examples  of  actions  and  outcomes 
that  are  best  characterized  by  one  of  these  three  categories. 
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Table  8.  Examples  of  three  categories  of  mariner  actions  or  outcomes. 


Category  of  Mariner  Action  or  Outcome 

Example  Descriptions 

Verbal  or  written  response  to  question 

Verbal  response  to  question:  What  are  the  duties 
and  responsibilities  of  the  lookout? 

Sequence  of  operational  actions 

Keep  the  victim  (person  overboard)  in  sight  at  all 
times 

Action  resulting  in  an  observable  outcome 

Report  sighted  objects  verbally  using  degrees 

Identify  performance  requirements  for  each  action  or  outcome.  The  second  step  is  to 
identify  the  performance  requirements  of  each  mariner  action  or  outcome.  Here,  a  performance 
requirement  is  an  aspect  or  characteristic  of  the  action  or  outcome  required  for  successful  job 
performance.  A  mariner  action  or  outcome  may  have  a  single  requirement  or  it  may  have  several 
requirements.  An  example  of  an  action  with  a  single  requirement  from  the  Lookout  assessment 
is  Call  the  bridge  if  you  are  not  properly  relieved.  The  single  requirement  is  to  notify  the  bridge. 
An  example  of  an  action  with  multiple  requirements  for  the  Lookout  assessment  is  Reports 
sighted  objects  verbally  using  degrees.  This  action  requires  the  lookout’s  reporting  (1)  a 
minimum  number  of  objects,  (2)  what  each  object  is,  (3)  where  each  object  is,  and  (4)  how  far 
away  each  object  is.  During  this  step,  separate  performance  requirements  will  be  identified. 
During  the  next  two  steps,  separate  measures  corresponding  to  each  requirement  will  be 
established. 

Select  instrumented  measures  when  available.  Because  instrumented  measures  are  inherently 
more  objective,  reliable,  and  free  of  observer  bias,  they  should  be  selected  for  use  in  assessments 
whenever  available.  In  the  present  Lookout  example  of  Reports  sighted  objects  verbally  using 
degrees,  the  use  by  the  assessor  of  a  bearing  repeater,  or  some  other  device  providing  accurate 
bearings,  is  indicated.  In  the  engine  room  environment,  many  actions  rely  on  the  reading  of 
calibrated  indicators  or  the  operation  of  calibrated  controls.  In  this  case,  the  instruments  are 
readily  available  to  the  assessor  to  support  many  of  the  performance  measures. 

Develop  observational  procedures  and  measures.  All  performance  measures  depend  upon 
observation  by  an  assessor.  Thus,  observational  procedures  should  be  specified  in  all  cases 
where  they  are  not  readily  understood  in  advance.  If  the  assessor  is  using  observation  based  on  a 
reading  from  an  instrument,  there  will  typically  be  standard  procedures  for  observation.  For 
example,  accurate  reading  of  many  indicators  requires  that  the  observer's  eyes  be  positioned 
directly  above  the  pointer  to  avoid  the  affects  of  parallax,  or  apparent  displacement.  In  addition, 
it  will  often  be  necessary  to  specify  the  instrumented  measure  to  be  used  by  the  assessor.  For 
example,  using  a  bearing  repeater  to  measure  performance  of  Reports  objects  verbally  using 
degrees  includes  the  recording  of  the  deviation,  in  degrees,  between  the  reported  and  actual 
bearing  of  the  target. 

Observational  measures  require  special  attention  in  the  development  of  observational  procedures. 
Here,  the  measure  is  actually  defined  on  the  basis  of  the  explicit  observational  procedures.  In  the 
case  of  verbal  or  written  questions,  this  will  commonly  involve  specifying  the  question  to  be 
asked  by  the  assessor.  For  example,  the  measure  for  the  Lookout  action  Identify  and  describe  the 
international  distress  signals  consists  of  the  question  asked  by  the  assessor  Identify  and  describe 
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six  international  distress  signals.  In  the  case  of  the  observational  measure  of  actions  or 
outcomes,  the  development  of  observational  procedures  requires  specifying  the  individual  steps 
or  characteristics  of  performance.  For  the  Lookout  example,  Reports  sighted  objects  verbally 
using  degree,  this  development  involves  specifying  the  assessor’s  procedure  for  observing  each 
identified  performance  requirement.  These  include  (1)  counting  the  number  of  objects  reported, 
(2)  noting  the  reported  and  actual  type  of  each  object  reported,  (3)  recording  the  reported  and 
actual  bearing  of  each  object,  and  (4)  noting  the  reported  and  actual  distance  of  each  object. 


Establish  a  performance  standard  for  each  measure.  Then,  it  is  necessary  to  establish  a 
performance  standard  for  each  measure.  The  standard  is  extremely  critical  to  the  effectiveness  of 
the  assessment  in  ensuring  the  proficiency  of  the  mariner.  In  developing  standards  for  STCW 
certification,  the  criterion  for  proficiency  specified  in  that  document  is  an  important  first  source. 
Other  sources  for  establishing  standards  are  the  same  ones  used  for  establishing  the  assessment 
objectives  and  for  breaking  them  into  components.  These  include  government  regulations, 
technical  manuals,  job  instructions,  textbooks,  and  task  analyses.  Finally,  appropriate  experts 
can  and  should  be  consulted.  The  standard  of  performance  for  each  measure  should  be 
consistent  with  important  sources  and  expert  opinion.  In  the  case  of  instrumented  measures,  this 
standard  should  reflect  the  instrument’s  unit  of  measurement.  As  an  example,  in  the  case  of  the 
Lookout  measure  of  reporting  the  sighted  object’s  bearing,  the  standard  was  established  that 
Reports  must  be  within  22.5  degrees  of  the  actual  bearing  of  targets.  The  experts  who 
established  this  standard  considered  the  consequences  of  inaccurate  lookout  reports.  Based  on 
that  consideration,  it  was  determined  that  a  report  that  directed  the  officer  of  the  watch  to  a  45 
degree  area  would  be  adequate  to  meet  the  requirement  for  a  full  appraisal  of  the  situation  and  of 
the  risk  of  collision  (from  Rule  5  of  the  Navigation  Rules). 


Establishing  performance  standards  for  observational  measures  follows  an  analogous  process  to 
that  for  instrumented  measures.  However,  in  this  case,  the  unit  of  measure  and  standard  are 
typically  defined  together.  For  oral  or  written  questions,  this  will  require  a  definition  of 
acceptable  answers.  For  example,  for  the  Lookout  measure  “Answers  question  Identify  and 
describe  six  international  distress  signals ,”  the  following  potential  answers  are  predefined;  and 
the  standard  of  having  the  candidate  correctly  recall  six  distress  signals  was  established  on  the 
basis  of  expert  review. 


Red  star  shells 
Flames  on  a  vessel 

Orange  background  black  ball  &  square 
“Mayday”  by  radio 
Dye  marker  (any  color) 

Square  flag  and  ball 
Radio-telephone  alarm 


Fog  horn  continuing  sounding 
Gun  fired  at  intervals  of  1  minute 
SOS 

Parachute  red  flare 

Code  flags  November  Charlie 

Wave  arms 

Position  indicating  radio  beacon 


•  Smoke 


When  the  observational  measure  is  based  on  the  assessor’s  observation  of  procedural  steps  or 
outcomes,  it  is  sometimes  much  more  difficult  to  predefine  all  alternative  mariner  actions. 
However,  when  possible,  guidance  should  be  provided  in  the  assessment  procedure.  For 
example,  in  establishing  the  standard  for  the  Lookout  measure  Report  how  far  off  the  sighted 
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object  is,  the  standard  is  correctly  reporting  using  one  of  four  alternative  statements  (hull-down, 
on  the  horizon,  hull-up,  or  close  aboard).  In  other  cases,  the  expertise  of  the  assessor  must  be 
relied  upon  to  determine  if  the  candidate’s  performance  meets  the  standard.  For  example,  the 
Lookout  assessment  requires  correct  reporting  of  what  type  of  object.  Here,  it  is  left  to  the 
assessor’s  expert  judgment  to  determine  if  the  lookout’s  report  provides  sufficient  information  to 
meet  the  proficiency  criterion  as  intended  in  STCW  and  other  sources. 

While  the  ISD  literature  describes  a  variety  of  approaches  to  developing  performance  standards, 
only  “pass/fail”  standards  are  described  here.  During  assessment  trials,  it  was  determined  that 
the  pass/fail  approach  was  the  most  practical  for  the  assessor  doing  the  over-the-shoulder 
assessment.  In  the  simplest  case,  there  will  be  a  single  pass/fail  standard  for  a  single  measure 
and  a  single  action.  Table  9  is  an  excerpt  from  the  Lookout  assessment  that  illustrates  such  a 
case.  Here,  if  six  or  more  correct  international  distress  signals  are  recalled,  the  candidate’s 
performance  meets  that  standard  for  Action  3.1,  or  “passes;”  if  fewer  than  six  are  recalled,  the 
candidate’s  performance  fails  to  meet  the  standard  for. 

Table  9.  Example  of  a  single  measure  and  standard  applied  to  an  assessment  action. 


Action 

Performance 

Measure 

Performance  Standard 

3.1  Identify 

and  describe  the 
international 
distress  signals 

Answers 

question: 

Identify  and 
describe  six 
international 
distress  signals. 

Identifies  and  describes  six  of  the  following: 

□  Red  star  shells 

□  Fog  horn  continuing  sounding 

□  Flames  on  a  vessel 

□  Gun  fired  at  intervals  of  1  minute 

□  Orange  background  black  ball  &  square 

□  SOS 

□  “Mayday”  by  radio 

□  Parachute  red  flare 

□  Dye  marker  (any  color) 

□  Code  flags  November  Charlie 

□  Square  flag  and  ball 

□  Wave  arms 

□  Radio-telephone  alarm 

□  Position  indicating  radio  beacon 

□  Smoke 

When  multiple  measures  and  standards  are  developed  for  a  single  action,  each  of  the  individual 
standards  must  be  met  to  pass  the  requirements  for  that  action.  Table  10  is  an  excerpt  from  the 
Lookout  assessment  that  applies  this  more  complex  scoring  procedure.  In  this  case,  the  candidate 
must  correctly  report  (1)  a  minimum  of  three  objects,  (2)  what  (type  of  object),  (3)  where  (within 
22.5  degrees),  and  (4)  how  far  off  (hull-down,  on  the  horizon,  hull-up,  or  close  aboard)  to  meet 
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the  set  of  multiple  standards. 


Table  10.  Example  of  multiple  standards  applied  to  a  single  action. 


Action 

Performance 

Measure 

Performance  Standard 

4.3  Report  sighted 
objects  verbally 
using  degrees 

Reports  sighted 
objects  verbally 
using  degrees 

□  Verbally  reports  at  least  3  surface  objects. 
Reports  all  visible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports 
must  be  within  22.5°  of  the  actual  bearing  of 
detected  targets. 

□  How  far-off  (hull-down,  on  the  horizon,  hull- 
up,  close  aboard). 

Failure  of  any  of  the  above  four  standards 

constitutes  failure. 

4.3  Develop  proficiency  criteria  and  scoring  procedures 

Mariner  proficiency  refers  to  an  individual’s  demonstrated  ability  to  meet  the  job  performance 
requirements.  Up  to  this  point  in  the  development  of  assessment  procedures,  job  requirements 
have  been  analyzed  to  identify  individual  assessment  objectives  and  these  have  been  further 
analyzed  into  component  actions.  Each  action  has  been  reviewed  to  identify  appropriate 
performance  requirements,  and  individual  measures  and  standards  have  been  developed.  For 
each  action,  a  pass/fail  standard  has  been  developed.  Now,  proficiency  criteria  should  be 
developed  for  each  assessment  objective  that  combine  the  standards  for  individual  actions  on  the 
basis  of  the  action’s  job  criticality.  Figure  2  outlines  the  general  process  for  developing 
proficiency  criteria  and  scoring  procedures.  This  process  involves  separate,  parallel  paths  for 
critical  and  non-critical  actions,  concluding  with  the  development  of  a  combined  proficiency 
criteria  and  scoring  procedure  for  a  single  assessment  objective.  The  following  discussion 
describes  this  process. 


Figure  2.  General  process  for  developing  proficiency  criteria  and  scoring  procedures. 
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Identify  critical  actions.  The  first  step  in  developing  proficiency  criteria  and  scoring 
procedures  requires  a  final  consideration  of  the  criticality  of  the  action  being  assessed.  The  key 
consideration  is  whether  the  action  is  essential  to  safe  and/or  effective  job  performance.  An 
example  of  a  critical  action  is  in  the  Demonstrate  man  overboard  procedures  objective  of  the 
Lookout  assessment.  Successfully  meeting  the  three  standards  of  (1)  establishes  and  maintains 
visual  contact,  (2)  points  in  the  direction  of  the  person  in  the  water,  and  (3)  remains  on  station 
until  he  or  she  is  properly  relieved  or  ordered  otherwise  are  all  considered  essential  to  safety. 

All  actions  under  a  single  assessment  objective  should  be  considered  individually  and  all  critical 
actions  identified. 

Establish  pass/fail  proficiency  criteria.  Because  critical  actions  are  essential  for  safe  and/or 
effective  job  performance,  a  pass/fail  proficiency  criterion  should  be  established  for  each  of 
these.  That  is,  failure  to  meet  the  standard(s)  corresponding  to  that  action  will  be  established  as 
sufficient  for  failing  the  proficiency  criteria  of  the  corresponding  assessment  objective. 
Continuing  with  the  Lookout  objective  Demonstrate  man  overboard  procedures,  failure  to  meet 
the  standards  for  the  action  Keep  the  victim  in  sight  at  all  times  was  established  as  sufficient  for 
failure  of  this  objective. 

Identify  non-critical  actions.  The  lower  path  in  Figure  2  is  taken  for  those  actions  that 
correspond  to  non-critical  job  requirements.  Here,  meeting  the  standards  for  each  action  is  not 
essential  to  operational  safety  and/or  effectiveness.  For  example,  everyone  is  familiar  with 
driver’s  license  tests  that  have  established  passing  rates,  such  as  80  percent.  In  these  cases,  the 
assessment  typically  addresses  a  procedure  that  includes  safety  redundancies  or  a  broad  area  of 
knowledge  where  general,  not  perfect,  understanding  and  comprehension  is  established  as  the 
target.  In  the  present  Lookout  example,  all  actions  were  determined  to  be  critical.  However, 
other,  non-critical  lookout  actions  could  be  identified  as  part  of  an  assessment  of  lookout 
proficiency. 

Establish  graded  proficiency  criteria.  If  a  related  set  of  non-critical  actions  has  been  identified 
for  an  assessment  objective,  it  may  be  most  appropriate  to  develop  one  or  more  sets  of  graded 
proficiency  criteria.  The  related  set  of  actions  should  be  reviewed  by  operational  experts,  who 
should  assess  the  risks  associated  with  failure  in  meeting  the  standards  associated  with  each 
action.  Then,  based  on  this  review,  a  proficiency  criterion  should  be  established  for  the  related 
set  of  actions.  If  more  than  one  set  of  non-critical,  related  actions  is  identified  within  a  single 
assessment  objective,  this  process  should  be  repeated  for  each  of  these  sets. 

As  noted  above,  the  present  Lookout  example  does  not  include  non-critical  actions.  However, 
subsequent  refinement  of  this  assessment  for  application  aboard  commercial  vessels  resulted  in 
the  combination  of  five  related  Lookout  assessment  objectives.  All  five  of  these  objectives  were 
judged  to  involve  interrelated  knowledge  and  understanding  requirements,  and  a  single 
proficiency  criterion  of  80  percent  correct  was  established  for  the  entire  set.  The  reader  can  refer 
to  these  related  assessment  procedures  in  McCallum,  Bames,  Forsythe,  and  Smith,  2000. 

Combine  pass/fail  and  graded  proficiency  criteria.  The  final  box  in  Figure  2  denotes  the 
development  of  a  proficiency  criterion  across  all  critical  and  non-critical  actions  for  the 
assessment  objective  under  consideration.  There  are  three  possible  strategies  that  must  be 
considered  at  this  point,  as  presented  by  the  hypothetical  examples  of  assessment  scores  and 
results  in  Table  1 1 .  If  only  critical  actions  have  been  identified,  then  a  corresponding 
proficiency  criterion  can  be  established  that  requires  the  candidate  to  receive  a  passing  score  on 
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all  actions  for  the  corresponding  assessment  objective.  The  assessment  objective  demonstrate 
man  overboard  procedures  is  such  an  example.  Here,  all  three  actions  were  determined  to  be 
critical.  Thus,  the  resulting  proficiency  criterion  for  this  assessment  objective  requires  the 
candidate  to  pass  the  individual  standards  corresponding  to  each  of  the  three  actions.  A 
hypothetical  example  of  this  case  is  shown  in  the  “All  Critical”  column  of  Table  11.  For  this 
case,  meeting  the  standards  for  eight  actions  corresponds  with  a  score  of  8  and  a  result  of  not 
meeting  the  proficiency  criterion  for  the  assessment  objective  (i.e.,  failing). 


Table  11.  Examples  of  hypothetical  assessment  scores  and  results. 


Assessment  Objective 

Criteria, 
Procedures 
,  &  Results 

Examples  with  Different  Criticality  of  Actions 

Action 

Score 

Pass  Fail 

All  Critical 

All  Non-critical 

Critical  &  Non- 

critical 

1. 

d  □ 

Proficiency 

Actions  1-10 

Actions  1-5 

Actions  1-5 

2. 

d  □ 

Criteria 

(critical):  100% 

(non-critical): 

80% 

(critical)  100% 

rrtf  1 — I 

Actions  6-10 

3. 

IS  U 

Actions  6-10 

(non-critical)  60% 

4. 

□  af 

(non-critical): 

C 

i/  □ 

60% 

O. 

Scoring 

Actions  1-10  = 

Actions  1-5  = 

Actions  1-5  = 

6. 

□  d 

Procedures 

80%  FAIL 

80%  PASS 

80%  FAIL 

7. 

Actions  6-10  = 

Actions  6-1 0  = 

8. 

a  □ 

80%  PASS 

80%  PASS 

9. 

d  □ 

Assessment 

Apply  criterion  for 

Apply  both  criteria 

Apply  both  criteria 

iw  n 

Result 

assessment 

for  assessment 

for  assessment 

10. 

LxJ  l — 1 

b(  □ 

objective 

objective 

objective 

FAIL 

PASS 

FAIL 

When  only  non-critical  actions  have  been  identified,  a  single  graded  proficiency  criterion  or 
combined  set  of  graded  proficiency  criteria  should  be  established.  If  a  single  set  of  interrelated 
actions  is  identified,  a  single  graded  proficiency  can  be  established.  However,  if  analysis  of  the 
set  of  actions  reveals  two  or  more  independent  sets  of  actions,  each  with  unrelated  implications 
of  failure,  then  two  independent  sets  of  graded  proficiency  criteria  should  be  established.  A 
hypothetical  example  of  the  latter  case  is  presented  in  the  “All  Non-critical”  column  of  Table  1 1 . 
For  this  case,  scores  of  80  percent  for  the  two  sets  of  Actions  1-5  and  Actions  6-10,  result  in 
meeting  both  criteria  for  the  assessment  objective. 

Finally,  when  both  critical  and  non-critical  actions  are  identified,  then  the  individual  sets  of 
actions  must  be  considered  separately  in  establishing  proficiency  criteria  and  scoring  procedures. 
The  last  column  in  Table  1 1  presents  this  example.  For  this  case,  Actions  1-5  are  critical  and 
require  passing  each  action;  and  Actions  6-10  are  non-critical,  requiring  a  60  percent  passing 
rate.  Combining  these  two  scoring  procedures  results  in  not  meeting  the  proficiency  criterion  for 
the  assessment  objective. 
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4.4  Validate  measures,  standards,  proficiency  criteria,  and  scoring  procedures 

The  proficiency  criteria  corresponding  to  each  assessment  objective  must  be  validated  in  a 
manner  that  ensures  that  assessment  will  be  conducted  consistently  and  represent  the 
requirements  of  the  job.  Proficiency  is,  in  the  end,  a  judgment  that  takes  into  account  both  the 
risks  associated  with  sub-optimal  performance  and  the  costs  of  requiring  optimal  performance  of 
all  mariners.  A  commonly  accepted  means  of  establishing  the  validity  of  an  assessment 
procedure  is  to  convene  a  meeting  of  experts  who  formally  review  and  reach  consensus  on  the 
validity  of  the  assessment.  Within  large  organizations,  it  is  often  most  efficient  to  establish  a 
standing  committee  to  review  and  validate  new  or  updated  assessment  procedures.  If  this 
approach  is  not  efficient,  it  may  be  best  to  establish  either  a  standing  or  ad  hoc  committee  from 
among  several  organizations. 

Validation  of  an  assessment  procedure  requires  that  experts  in  the  relevant  proficiency 
independently  review  each  of  the  assessment  development  steps  up  to  this  point,  discuss  their 
areas  of  concern,  and  reach  agreement  that  the  final  assessment  procedure  will  provide  a  reliable 
and  valid  means  of  assessing  mariner  proficiency.  The  validation  process  can  become  time- 
consuming  and  inefficient  if  it  is  not  managed  effectively.  However,  if  it  is  managed  effectively 
and  all  critical  issues  are  considered  and  agreed  upon,  the  resulting  assessment  procedure  will 
have  a  much  greater  and  more  enduring  value.  Refer  to  the  reference  section  for  texts  that 
address  procedures  for  validation  (e.g.,  Jonassen,  Hannum,  and  Tessmer,  1989;  Rothwell  and 
Kazanas,  1998).  Basic  requirements  of  an  adequate  validation  process  based  on  committee 
review  are  summarized  below: 

•  Assemble  a  committee  of  at  least  three  members  who  are  fully  qualified  and  experienced 
in  the  area  of  proficiency  under  consideration. 

•  Ask  the  committee  to  discuss  and/or  observe  performance  of  the  job  and  tasks  associated 
with  the  assessment  objectives  in  both  realistic  work  settings  and  the  proposed 
assessment  conditions. 

•  Ask  each  expert  to  consider  and  comment  independently  on  the  acceptability  of  the 
assessment  objectives,  conditions,  measures,  proposed  standards,  and  proposed 
proficiency  criteria. 

•  Ask  the  committee  members  to  review  and  discuss  issues  until  agreement  among  all 
members  of  the  group  has  been  established. 

A  final  recommended  step  in  validation  is  trial  application  of  the  procedure.  Here,  draft  versions 
of  the  assessment  are  used  on  a  trial  basis  to  identify  issues  that  would  not  be  recognized 
otherwise.  This  final  step  is  addressed  in  the  final  discussion  of  implementation  issues  that 
follows. 

5.0  Prepare  the  Assessment  Materials 

Goal  of  this  step 

Prepare  all  materials  and  tools  so  that  they  are  ready  for  use  by  an  assessor  and  candidate. 
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Sub-steps 

5. 1  Prepare  detailed  assessment  worksheets. 

5.2  Prepare  assessor  instructions. 

5.3  Prepare  candidate  instructions. 

5.4  Finalize  written  and  oral  test  questions. 

5.5  Finalize  simulator  and/or  shipboard  procedures. 

5.6  Document  references,  source  materials,  and  validation  process. 

5.1  Prepare  detailed  assessment  worksheets 

The  assessment  materials  must  be  prepared  as  stand-alone  procedures  that  can  be  administered 
by  assessors  who  are  not  familiar  with  the  development  process.  This  will  help  to  yield  valid  and 
reliable  assessment  results.  The  central  assessment  materials  are  the  written  procedures  or 
worksheets  that  will  be  used  by  the  assessor  while  conducting  the  assessment.  These  worksheets 
should  be  prepared  first,  since  they  will  likely  be  referred  to  in  all  of  the  other  assessment 
materials.  The  present  Lookout  example  has  a  series  of  seven  worksheets,  each  of  which 
corresponds  to  one  or  more  assessment  objectives.  Each  worksheet  is  formatted  into  three 
sections,  consisting  of  (1)  identifying  information,  (2)  scoring  and  recording  procedures,  and  (3) 
comments  that  provide  succinct  instructions  for  the  assessor. 

The  assessment  worksheet  should  provide  sufficient  information  to  guide  the  assessor  through 
the  assessment  procedure,  as  well  as  a  means  of  recording  the  outcome  corresponding  to  the 
individual  objectives,  objective  components,  measures,  and  standards.  It  should  be  noted  that 
detailed  assessor  instructions  could  take  the  place  of  highly  elaborate  assessment  worksheets. 

That  is,  much  of  the  information  required  for  conducting  an  assessment  can  be  included  in  the 
assessor  instructions,  with  only  the  minimum  necessary  for  recording  the  results  of  the 
assessment  included  in  the  worksheet.  On  the  other  hand,  assessor  instructions  can  be  limited  to 
the  necessary  information,  with  more  detailed  guidance  provided  directly  on  the  worksheet.  This 
choice  should  be  based  on  a  consideration  of  the  amount  of  assessment  training  available  to 
assessors  and  the  amount  of  information  that  can  be  reasonably  included  in  these  two  documents, 
among  other  considerations. 

5.2  Prepare  assessor  instructions 

Every  set  of  assessment  materials  should  include  a  set  of  assessor  instructions.  In  many  cases, 
detailed  step-by-step  instructions  help  to  ensure  the  reliability  and  validity  of  the  resulting 
assessment.  Although  proficient  mariners  should  be  conducting  the  assessment,  there  can  be 
many  interpretations  of  how  an  assessment  is  to  be  conducted,  especially  with  regard  to 
candidate  instructions,  equipment  set-up,  timing  and  pace  of  assessment  activities,  and 
interaction  between  the  assessor  and  candidate. 

The  assessor  instructions  could  include  the  following  topics: 

•  Prepare  for  assessment. 

•  Conduct  pre-briefing  with  the  candidate. 
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•  Observe  performance  and  record  results. 

•  Determine  assessment  outcome. 

•  Conduct  assessment  debriefing. 

•  Special  assessment  instructions. 

The  Lookout  assessment  includes  an  assessor  instructions  checklist  (see  the  assessment 
procedure  at  the  back  of  this  manual)  that  provides  detailed  guidance  on  conducting  the  Lookout 
assessment,  following  the  topics  outlined  above. 

5.3  Prepare  candidate  instructions 

The  set  of  assessment  materials  should  also  include  separate,  self-contained  candidate 
instructions  that  can  be  provided  to  the  candidate  well  in  advance  of  when  the  assessment  is 
conducted.  At  a  minimum,  the  candidate  instructions  should  include: 

•  Assessment  objective  source  -  STCW Code  chapter,  table,  function,  competence,  and 
proficiency. 

•  Assessment  objective(s). 

•  Assessment  method(s). 

•  Conditions  under  which  the  assessment  will  be  conducted  (equipment  status,  underway 
versus  in  port,  etc.). 

•  Assessment  procedures,  in  terms  of  who  will  administer  the  assessment  and  how  long  it 
will  take. 

•  A  summary  of  the  measures  and  standards  that  will  be  used  to  determine  the  outcome  of 
the  assessment. 

The  Lookout  assessment  includes  a  set  of  candidate  instructions  (see  the  assessment  procedure  at 
the  back  of  this  manual)  that  address  what  will  take  place  during  the  pre-assessment  briefing, 
performance  observation  and  recording,  and  determination  of  the  assessment  outcome. 

5.4  Finalize  oral  and  written  questions 

As  noted  earlier  in  the  discussion  of  sub-step  3.2  -  Specify  oral  and  written  questions,  written 
items  should  be  documented  by  either  defining  detailed  question  objectives  or  developing  a 
library  of  questions.  The  assessment  developer  must  be  aware  of  the  dangers  of  inadequate 
specification  of  oral  and  written  questions.  Without  a  detailed  set  of  questions,  the  performance 
measures  are  not  defined  and  any  performance  standards  are  arbitrary.  At  a  minimum,  the 
specific  knowledge  areas  to  be  assessed  should  be  defined,  along  with  the  format  of  the 
assessment  items  (i.e.,  true/false,  multiple-choice,  short  answer,  or  essay/discussion)  and  the 
number  of  items  that  should  be  included  for  each  knowledge  area.  The  developer  should 
consider  providing  a  detailed  list  of  specific  oral  and/or  written  questions,  along  with  the  correct 
answers  for  each  question. 
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5.5  Finalize  simulator  and/or  shipboard  procedures 

Just  as  it  is  necessary  to  detail  written  and  oral  test  questions,  detailed  procedures  for  simulator 
and/or  shipboard  assessment  procedures  should  also  be  finalized  in  the  assessment  materials. 
When  simulators  are  used,  specific  assessment  scenarios  or  scripts  should  be  developed.  In 
addition  to  performance  measurement  and  recording  procedures,  fully  documented  simulator 
scripts  or  scenarios  might  include  the  following  specifications: 

•  Simulated  time  of  day  and  weather  conditions. 

•  Own  vessel  status  at  the  start  of  the  scenario. 

•  Other  vessel  locations  at  the  start  of  the  scenario. 

•  Sequence  and  timing  of  own  vessel  malfunctions,  if  any. 

•  Course,  speed,  and  maneuvers  of  other  vessels,  if  any. 

In  the  case  of  shipboard  assessment,  a  special  concern  is  the  need  to  develop  a  detailed  plan  for 
the  assessment.  It  will  be  necessary  to  find  a  balance  between  controlling  and  selecting  the 
conditions  of  assessment.  The  shipboard  assessment  procedures  should  detail  both  what 
conditions  the  assessor  should  establish  in  advance  of  the  assessment  and  what  conditions  must 
be  taken  advantage  of  or  avoided  when  conducting  an  assessment  during  normal  operations. 
Documentation  of  shipboard  assessment  procedures  should  include  a  worksheet  that  provides 
space  and  instructions  for  the  assessor  regarding  the  following  items,  as  applicable: 

•  Critical  safety  issues. 

•  Equipment  initialization  and  set-up. 

•  Recording  of  own  vessel  conditions  (date,  time,  location,  course,  speed,  equipment 
status). 

•  Environmental  conditions. 

•  Traffic  conditions. 

•  Cues  to  be  provided  to  the  candidate  by  the  assessor  or  equipment. 

•  Performance  to  be  observed  and  recorded. 

5.6  Document  references,  source  materials,  and  the  validation  process 

The  final  step  in  preparing  the  assessment  materials  is  to  document  references,  source  materials, 
and  the  validation  process.  This  last  step  serves  two  purposes.  First,  it  provides  the  necessary 
references  and  documentation  to  allow  either  the  assessment  developer,  or  a  successor,  to 
maintain  and  update  the  assessment  procedure.  Second,  this  documentation  provides  necessary 
reference  material  to  assist  an  external  reviewer  who  may  evaluate  the  assessment  procedure. 

Implementation 

When  all  assessment  materials  are  prepared,  the  assessment  procedure  is  ready  for 
implementation.  Implementation  may  be  as  straightforward  as  handing  the  document  over  to  the 
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assessor  to  begin  use  in  assessing  candidates.  Or  implementation  may  involve  a  more  complex 
process  that  requires  several  iterations  of  trial  application  and  refinement  before  the  procedures 
are  ready  to  serve  their  function  in  mariner  assessment.  More  than  likely,  some  intermediate 
level  of  trial  application  and  refinement  will  be  required  prior  to  full  implementation. 
Additionally,  assessment  developers  and  assessors  must  remain  cognizant  of  changes  in 
operational  equipment,  procedures,  or  regulatory  requirements  that  will  necessitate  future 
revisions  to  the  implemented  assessment  procedures.  Issues  concerning  the  implementation  of 
assessment  procedures  and  the  conduct  of  assessments  are  addressed  in  a  companion  manual, 
Conducting  Mariner  Assessments  (McCallum,  Barnes,  Forsythe,  &  Smith,  2000). 
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GLOSSARY 


Assessor.  Anyone  who  conducts  an  assessment  or  evaluation  of  an  individual’s  proficiency. 
Although  the  use  of  the  term  assessor  is  retained  in  some  discussions  of  STCW  requirements 
including  NVIC  4-97  on  company  roles  and  responsibilities,  the  term  designated  examiner  is 
used  for  assessor  in  the  U.S.  implementing  regulations. 

Assessment.  The  process  of  evaluating  whether  an  individual’s  performance  meets  established 
proficiency  criteria.  The  terminology  used  for  this  process  in  the  U.S.  implementing  regulations 
includes  both  an  examination  for  knowledge  and  an  assessment  based  on  a  practical 
demonstration  as  witnessed  by  a  designated  examiner. 

Assessment  conditions.  The  assessment  conditions  define  the  setting,  tools,  references,  aids, 
and  safety  precautions  that  are  in  place  at  the  time  that  a  candidate’s  proficiency  is  assessed. 

Assessment  objective.  A  critical  requirement  of  job  performance  that  can  be  measured  and 
assessed.  Assessment  objectives  include  skills  and  knowledge  required  by  the  job.  A  complete 
assessment  objective  description  includes  the  required  mariner  performance,  the  conditions  of 
assessment,  and  the  standards  of  performance  for  successful  accomplishment  of  the  objective. 

Assessment  procedures.  The  activities  that  are  conducted  in  administering  the  assessment  of  a 
candidate’s  proficiency.  The  term  assessment  procedure  can  be  used  to  describe  either  the 
actions  taken  or  the  written  instructions  and  activity  descriptions  that  are  carried  out  in 
conducting  an  assessment. 

Designated  examiner.  A  person  who  has  been  trained  or  instructed  in  techniques  of  training  or 
assessment  and  is  otherwise  qualified  to  administer  performance  assessment  procedures,  in 
accordance  with  USCG  requirements.  Further  details  on  the  qualifications  of  designated 
examiner  can  be  found  in  NVIC  6-97. 

Instrumented  measure.  A  performance  measure  that  relies  upon  apparatus  that  can  be 
calibrated  to  yield  highly  consistent  and  accurate  measurement  results. 

Job.  A  post  of  employment  consisting  of  a  cluster  of  related  work  responsibilities  and  duties 
(e.g.,  chief  engineer,  third  mate,  able-bodied  seaman).  In  the  STCW  Code,  a  job  is  further 
defined  on  the  basis  of  licensure  level  (e.g.,  officer  in  charge  of  a  navigational  watch  on  ships  of 
500  gross  tonnage  or  more). 

Knowledge.  The  learned  concepts,  cues,  facts,  rules,  and  procedures  that  are  necessary  for 
proficient  performance  of  a  task  (e.g.,  knowledge  of  algebra,  knowledge  of  the  Navigation  Rules, 
knowledge  of  procedures  for  starting  the  main  engine). 

Observational  measure.  A  performance  measure  that  relies  primarily  upon  an  assessor's 
observation  and  interpretation  of  mariner  performance  to  determine  the  assessment  outcome. 

Performance  measure.  The  procedures  used  by  the  assessor  in  observing  and  recording 
mariner  actions,  or  the  outcome  of  those  actions.  The  effectiveness  of  the  assessment  depends 
on  the  care  with  which  these  procedures  are  developed  and  followed. 

Performance  standard.  The  acceptable  or  target  level  established  for  individual  performance 
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measures.  The  assessor  must  interpret  observed  performance  as  passing  or  failing  the 
established  standard. 

Proficiency.  An  individual’s  demonstrated  ability  to  meet  selected  job  performance 
requirements,  as  established  on  the  basis  of  performance  measures,  performance  standards,  and 
proficiency  criteria. 

Proficiency  criteria.  A  rule  established  for  combining  the  measures  and  standards  for 
individual  actions  and  providing  a  result  for  the  larger  unit  of  performance  encompassed  by  the 
assessment  objective.  Proficiency  criteria  can  be  pass/fail  to  critical  actions  or  graded  for  non- 
critical  actions. 

Qualified  instructor.  According  to  the  United  States  implementing  regulations,  “the  person 
who  has  been  trained  or  instructed  in  instructional  techniques  and  is  otherwise  qualified  to 
provide  required  training  to  candidates  for  licenses,  documents,  or  endorsements.”  Further 
details  on  the  qualifications  of  qualified  instructors  can  be  found  in  NVIC  6-97. 

Reliability.  The  consistency  of  a  measurement  procedure.  In  the  context  of  assessment, 
reliability  generally  can  be  defined  as  the  consistency  of  the  assessment  outcome  when  applied 
under  comparable  conditions.  The  reliability  of  an  assessment  establishes  the  maximum  level  of 
possible  assessment  validity.  That  is,  an  assessment  can  not  be  any  more  valid  than  it  is  reliable. 

Scoring  procedure.  The  defined  procedures  for  combining  individual  performance  measures 
and  performance  standards  in  the  application  of  proficiency  criteria. 

Skills  and  abilities.  The  behaviors  that  must  be  applied  in  successful  performance  (e.g.,  typing 
skills,  equipment  fault-finding  skills,  navigation  skills,  shiphandling  skills).  Measurable  and 
observable  skills  are  those  of  interest  in  proficiency  assessment. 

Task.  A  single,  observable  work  assignment  that  is  independent  of  other  actions  and  results  in  a 
valuable  outcome.  A  task  must  be  observable,  be  a  complete  work  assignment,  have  a  specific 
beginning  and  end,  and  be  able  to  be  measured  by  its  intended  product  or  outcome. 

Validity.  In  the  context  of  assessment,  validity  can  be  defined  as  the  degree  to  which  an 
assessment  accurately  reflects  the  skill,  knowledge,  and  performance  requirements  of  the  job. 
The  maximum  validity  of  an  assessment  is  established  on  the  basis  of  its  reliability.  That  is,  an 
assessment  can  not  be  any  more  valid  than  it  is  reliable. 
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Appendix  A  to 

“A  Method  for  Developing  Mariner  Assessments” 


MARINER  ASSESSMENT  PROCEDURES  FOR  THE 
PERFORMANCE  OF  LOOKOUT  DUTIES 


This  appendix  contains  example  procedures  for  assessing  a 
mariner’s  ability  to  perform  lookout  duties.  The  assessment 
package  consists  of  assessor  instructions,  candidate  instructions, 
seven  assessment  worksheets  and  an  assessment  control  sheet  for 
documenting  the  final  results  of  the  assessment. 


This  report  maybe  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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Assessment  Procedures/Lookout  Duties 


ASSESSOR  INSTRUCTIONS 


Introduction 

The  following  procedures  are  designed  for  assessing  a  candidate’s  ability  to  perform  lookout 
duties.  The  assessment  includes  demonstrating  lookout  techniques  and  making  lookout  reports 
in  restricted  visibility  and  in  clear  visibility  during  daylight  and  at  night. 

The  assessment  materials  include  Assessor  Instructions,  Candidate  Instructions,  an  Assessment 
Control  Sheet,  and  seven  separate  assessment  modules  (Assessment  Worksheets  I- VII).  The 
Assessor  Instructions  describe  the  assessment  objectives,  method,  performance  measures,  and 
performance  standards.  The  Assessor  Instructions  also  list  the  responsibilities  of  the  assessor 
during  each  phase  of  the  assessment  process.  The  Candidate  Instructions  explain  the  assessment 
process  from  the  candidate’s  perspective  and  define  the  responsibilities  of  the  candidate. 

The  Assessment  Control  Sheet  provides  a  record  of  the  names  of  the  candidate,  assessor,  and 
qualified  instructor  who  certified  that  the  candidate  met  the  training  prerequisites  for  assessment. 
The  control  sheet  also  includes  the  date  and  location  of  the  assessment  and  the  STCW  reference 
information  for  the  competency  area  being  assessed.  Section  3  of  the  Assessment  Control  Sheet 
provides  space  for  the  assessor  to  record  the  candidate’s  score  on  each  assessment  objective. 
Finally,  section  4  of  the  control  sheet  summarizes  the  assessment  objectives  and  the  actions 
comprising  each  objective. 

As  the  assessor,  you  will  evaluate  the  candidate’s  ability  to  complete  the  assessment  objectives 
correctly.  Using  the  scoring  procedures  described  below,  you  will  provide  the  candidate  with  a 
separate  score  for  each  assessment  objective  and  compile  these  scores  into  a  final  score  for  the 
entire  lookout  assessment.  The  scores  for  each  assessment  objective  and  the  candidate’s  overall 
score  for  the  entire  assessment  are  to  be  recorded  on  the  Assessment  Control  Sheet. 

The  rest  of  this  section  briefly  describes  the  various  components  of  this  assessment.  Assessment 
Worksheets  I  through  VII  provide  additional  information  about  the  assessment  objectives, 
method,  conditions,  performance  measures  and  performance  standards. 

Assessment  Objectives 

During  this  assessment,  the  candidate  should  demonstrate  his  or  her  ability  to: 

1 .  Describe  lookout  duties  and  responsibilities. 

2.  Identify  lookout  stations  and  safe  routes  onboard. 

3.  Describe  and  identify  the  international  distress  signals. 

4.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  during 
daylight. 

5.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  at  night. 

6.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  restricted  visibility  during 
daylight  or  at  night. 
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7.  Demonstrate  the  use  of  lookout  equipment. 

8.  Demonstrate  man  overboard  procedures. 

9.  Demonstrate  lookout  watch  relief  procedures. 

Assessment  Method 

Although  candidates  can  demonstrate  their  knowledge  of  the  lookout  duties  in  a  written  test,  to 
ensure  proficiency  in  the  actual  operating  environment,  candidates  should  undergo  assessment 
either  in  a  bridge  simulator  or  onboard  ship.  This  assessment  package  can  be  applied  using 
either  method. 

Assessment  Conditions 

Candidates  should  be  posted  at  a  lookout  station  equipped  with  an  internal  communications 
system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and  bearing  repeater  fitted  with  a 
bearing/azimuth  circle,  alidade,  or  pelorus.  The  lookout  station  should  be  clear  and  the  assessor 
must  be  able  to  observe  activities. 

The  conditions  for  Assessment  Worksheets  IV,  V  and  VI  are  as  follows: 

•  IV  -  Clear  visibility  during  daylight. 

•  V  -  Clear  visibility  at  night. 

•  VI  -  Restricted  visibility  during  daylight  or  at  night. 

The  conditions  for  the  other  assessment  modules  (worksheets  I,  II,  III,  and  VII)  can  be  any  of  the 
above. 

Performance  Measures  and  Standards 

Each  assessment  objective  is  comprised  of  one  or  more  actions.  Each  action  has  one  or  more 
corresponding  performance  measures  and  standards.  For  example,  assessment  objective  7 
requires  five  different  actions  (see  section  2  of  Assessment  Worksheet  V  on  p.  16).  Note  that 
within  one  action  there  may  be  more  than  one  performance  measure.  For  example,  in  the  case  of 
action  7.3,  the  candidate  has  to  pass  both  measures  in  order  to  pass  the  action.  If  a  candidate 
fails  one  measure,  he  or  she  fails  the  action. 

Note  that  if  a  candidate  fails  an  assessment  objective,  he  or  she  automatically  fails  the  entire 
assessment.  If  this  occurs,  you  should  terminate  the  assessment  without  proceeding  to  the  next 
objective. 

Assessment  Checklist 

The  following  checklist  summarizes  the  tasks  and  responsibilities  of  the  assessor  at  each  phase  of 
the  assessment  process. 
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Preparing  for  the  Lookout  Assessment 

□  Gather  Lookout  assessment  materials. 

□  Prepare  the  assessment  area(s),  test  facility,  and/or  lookout  station. 

□  Prepare  and  arrange  for  necessary  lookout  equipment;  ensure  that  all  equipment  is 
operational. 

□  If  you  are  using  a  simulator,  check  all  exercise  scenarios  to  ensure  that  they  include  all 
necessary  assessment  conditions. 

□  If  you  are  conducting  a  shipboard  assessment,  ensure  that  objects  are  present  for  the 
candidate  to  observe  and  report. 

□  Identify  and  observe  necessary  safety  precautions. 

□  Schedule  assessment  and  inform  all  affected  personnel. 

□  Review  the  Assessment  Control  Sheet,  Assessment  Worksheets,  and  Candidate  Instructions. 

Briefing  the  Candidate  before  the  Assessment 

□  Provide  the  candidate  with  a  copy  of  the  Candidate  Instructions  and  Assessment  Control 
Sheet. 

□  Review  the  lookout  assessment  instructions  with  the  candidate  and  answer  any  questions. 

□  Discuss  the  assessment  objectives  and  explain  any  that  require  the  use  of  non-standard 
procedures,  such  as  the  use  of  lookout  reporting  by  degrees  if  the  standard  procedure  onboard 
your  vessel  is  by  points. 

□  Explain  the  lookout  assessment  performance  measures  and  standards. 

□  Advise  the  candidate  of  the  time  constraints,  including  when  the  assessment  will  begin  and 
under  what  circumstances  the  assessment  will  be  terminated. 

□  Inform  the  candidate  that  he  or  she  should  use  appropriate  marine  terminology  and 
nomenclature  during  all  phases  of  the  assessment. 

□  Discuss  the  desired  outcome(s)  and  consequences  of  failing  to  perform  part  or  all  of  the 
lookout  assessment. 

□  Remind  the  candidate  that  it  is  permissible  to  ask  questions  during  the  assessment,  especially 
if  he  or  she  has  a  safety  concern. 

□  Inform  the  candidate  that  he  or  she  must  have  vision  correctable  to  at  least  20/40  in  each  eye 
and  uncorrected  vision  of  at  least  20/200  in  each  eye.  His  or  her  color  sense  must  be 
determined  to  be  satisfactory,  without  the  use  of  color-sensing  lenses. 

□  Inform  the  candidate  that  all  phases  of  the  assessment  are  to  be  conducted  in  English. 

□  Discuss  the  candidate’s  willingness  to  be  assessed  under  the  circumstances  presented. 
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Observing  the  Candidate’s  Performance 

□  If  a  safety  violation  occurs,  terminate  the  assessment  immediately. 

□  Continuously  observe  the  candidate  during  the  assessment.  Require  that  standard  procedures 
or  company  policy  be  adhered  to,  except  when  assessment  procedures  require  demonstration 
of  knowledge  or  skill  different  from  a  convention  adopted  by  the  company  (e.g.,  use  of 
degrees  to  report  sightings  when  points  are  the  convention). 

□  Ensure  realistic  assessment  conditions  consistent  with  a  normal  working  environment  for  the 
lookout.  Do  not  allow  the  candidate  to  “learn  the  test”  by  observing  the  performance  of  other 
assessment  candidates. 

□  Avoid  giving  the  candidate  unsolicited  assistance,  but  respond  to  appropriate  questions  and 
provide  appropriate  equipment  when  required. 

□  Remain  objective  and  maintain  positive  control  of  the  assessment  process  at  all  times. 

Recording  Results  and  Determining  Assessment  Outcome 

□  Record  performance  on  the  appropriate  lookout  assessment  worksheet. 

□  Strictly  adhere  to  the  prescribed  scoring  procedure(s). 

□  Determine  and  document  the  outcome  of  the  assessment,  then  transfer  the  final  results  to  the 
Assessment  Control  Sheet. 

□  Attest  to  successful  demonstration  of  tasks  in  the  Training  Record  Book  (TRB),  if 
appropriate. 

Debriefing  the  Candidate 

□  Debrief  the  candidate  as  soon  as  possible  after  the  assessment. 

□  Provide  the  candidate  with  a  copy  of  the  Assessment  Control  Sheet. 

□  Restate  the  assessment  objectives. 

□  Focus  on  positive  accomplishments  first. 

□  Identify  areas  needing  improvement. 

□  If  the  candidate  failed  to  demonstrate  proficiency,  jointly  develop  an  improvement  plan  to 
prepare  for  reassessment. 
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CANDIDATE  INSTRUCTIONS 


In  this  assessment,  you  will  be  evaluated  on  your  ability  to  perform  lookout  duties.  Table  1 
shows  the  1995  STCW  Code  specification  for  the  performance  of  lookout  duties. 


Table  1.  STCW  Code  specification  for  the  performance  of  lookout  duties. 


STCW  Requirement 

1995  STCW  Code,  Section  A-ll/4  (p.  68):  Mandatory  minimum  requirements 
for  certification  of  ratings  forming  part  of  a  navigational  watch. 

STCW  Function 

Navigation  at  the  support  level. 

STCW  Competence 

Keep  a  proper  lookout  by  sight  and  hearing. 

STCW  Proficiency 

Responsibilities  of  a  lookout,  including  reporting  the  approximate  bearing  of 
a  sound  signal,  light  or  other  object  in  degrees  or  points. 

Assessment  Objectives  and  Method 

Your  assessment  will  occur  either  onboard  ship  while  underway  or  in  a  bridge  simulator.  A 
qualified  assessor  will  assess  you  on  your  ability  to  meet  the  nine  assessment  objectives  in  the 
first  column  of  Table  2.  Note  that  each  assessment  objective  has  a  corresponding  assessment 
method.  Performance  measured  by  “oral  or  written  questions”  means  your  assessor  will  ask  you 
questions  pertinent  to  the  objective  and  assess  your  ability  to  respond  correctly  to  each  question. 
Performance  measured  by  “demonstration”  means  your  assessor  will  ask  you  to  demonstrate 
your  ability  to  perform  the  actions  required  in  the  objective. 


Table  2.  Assessment  objectives  for  lookout  duties. 


Assessment  Objective 

Assessment  Type 

1 .  Describe  lookout  duties  and  responsibilities 

Oral  or  written  questions 

2.  Identify  lookout  stations  and  safe  routes  on  board 

Oral  or  written  questions 

3.  Describe  and  identify  the  international  distress  signals 

Oral  or  written  questions 

4.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear 

visibility  during  daylight 

Practical  demonstration 

5.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear 

visibility  at  night 

Practical  demonstration 

6.  Demonstrate  lookout  techniques  and  make  lookout  reports  in 
restricted  visibility  during  daylight  or  at  night 

Practical  demonstration 

7.  Demonstrate  the  use  of  lookout  equipment 

Practical  demonstration 

8.  Demonstrate  man  overboard  procedures 

Practical  demonstration 

9.  Demonstrate  lookout  watch  relief  procedures 

Practical  demonstration 
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Refer  to  section  4  of  the  Assessment  Control  Sheet  for  a  list  of  the  specific  actions  required  for 
each  objective  (you  should  receive  a  copy  of  this  document  from  your  assessor  during  your  pre¬ 
assessment  briefing).  Below  are  some  general  guidelines  for  what  you  should  expect  during  the 
assessment  process. 

Pre-Assessment  Briefing  with  Assessor 

Your  pre-assessment  briefing  should  occur  approximately  a  week  in  advance  of  the  scheduled 
assessment,  if  possible.  [Note:  This  may  not  be  practical  or  feasible  for  many  of  the  trial 
implementations  conducted  for  research  purposes.]  This  will  help  you  and  the  assessor  to  be 
well  prepared  for  the  assessment  when  it  actually  occurs.  During  this  briefing,  you  should: 

•  Discuss  your  prior  experience,  training,  and/or  company  policy  with  the  assessor.  On  the 
basis  of  these  qualifications,  discuss  whether  you  are  qualified  to  undertake  the  lookout 
assessment.  If  you  both  agree  you  are  qualified,  then  continue  with  the  assessment 
process.  If  not,  arrange  for  additional  on-the-job  or  simulator  training,  and  set  a  date  for 
another  review  of  your  qualifications. 

•  Obtain  a  copy  of  the  Assessment  Control  Sheet  from  your  assessor.  Review  it  and  discuss 
the  scope  and  depth  of  knowledge  covered  by  the  Lookout  assessment.  Ask  questions 
about  any  part  of  the  assessment  that  is  unclear. 

•  Review  the  Lookout  assessment  performance  measures  and  standards,  and  ask  any 
questions  you  have  about  them. 

•  Discuss  the  desired  outcome(s)  and  the  consequences  of  failing  to  perform  any  part  of  the 
Lookout  assessment. 

•  Discuss  the  general  Lookout  assessment  procedures.  Your  assessor  will  inform  you  of 
how  much  time  is  allowed,  when  the  assessment  begins,  and  under  what  circumstances 
he  or  she  will  terminate  the  assessment. 

•  Discuss  any  non-standard  procedures  that  are  expected  of  you  during  the  assessment.  For 
example,  on  some  ships,  standard  procedure  is  to  report  objects  by  use  of  bells  or  points, 
but  during  the  lookout  assessment  you  must  also  report  objects  using  degrees. 

•  Consider  whether  you  are  willing  to  be  assessed  under  the  circumstances  presented  and 
advise  the  assessor  of  your  willingness  to  undertake  the  assessment. 

Participation  in  the  Assessment 

Your  assessor  will  continuously  observe  you  during  the  assessment.  You  are  expected  to  adhere 
to  standard  procedures  or  company  policy  unless  the  assessor  briefs  you  on  a  requirement  to 
perform  a  non-standard  procedure.  During  the  assessment,  remember  to: 

•  Use  appropriate  marine  terminology  and  nomenclature  at  all  times. 
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•  Listen  to  an  entire  question  before  responding  or  acting.  Remember  that  you  may  not  use 
reference  material  of  any  kind  during  the  oral  and  written  questions  portion  of  the 
assessment. 

•  Ask  questions  if  you  have  a  safety  concern.  The  assessor  will  provide  you  with 
appropriate  responses  to  your  questions.  If  a  safety  violation  occurs,  the  assessor  will 
terminate  the  assessment  immediately. 

Your  assessor  will  ensure  realistic  assessment  conditions  consistent  with  the  normal  working 
environment  for  a  lookout. 

The  Outcome  of  Your  Assessment 

The  assessor  will  record  your  performance  on  a  series  of  Assessment  Worksheets.  He  or  she  will 
score  each  assessment  objective  on  a  “Pass/Fail”  basis.  Acceptable  performance  (a  “Pass”  score) 
will  be  based  on  your  ability  to  correctly  respond  to  examination  questions.  It  will  also  be  based 
on  your  ability  to  perform  assigned  tasks  safely  in  a  manner  that  demonstrates: 

•  The  required  level  of  skill,  knowledge,  and  ability. 

•  Sound  and  professional  judgment. 

Unacceptable  performance  (a  “Fail”  score)  will  be  based  on  your  failure  to  perform  a  critical 
phase  of  the  assessment  exercise  at  an  acceptable  level  of  proficiency.  When  this  occurs,  your 
assessment  will  be  suspended  and  postponed  until  you  receive  further  instruction  and  training. 

Note  that  the  following  events  will  terminate  an  assessment  immediately: 

•  An  action,  or  lack  of  action,  by  you  which  required  corrective  action  or  intervention  by 
the  assessor  to  prevent  injury,  damage,  or  the  development  of  a  hazardous  condition. 

•  Your  failure  to  use  proper  procedures,  including  appropriate  communication  procedures, 
during  the  assessment. 

•  Your  failure  to  take  prompt  corrective  action  when  required. 

The  assessor  will  strictly  adhere  to  pre-determined  scoring  procedures,  and  will  document  the 
outcome  of  the  assessment.  He  or  she  will  transfer  the  final  results  of  your  assessment  to  the 
Assessment  Control  Sheet. 

Assessment  Debriefing 

You  and  the  assessor  should  discuss  the  assessment  results  as  soon  as  possible  after  the 
assessment.  During  the  debriefing: 

•  The  assessor  should  provide  you  with  a  copy  of  the  Assessment  Control  Sheet  describing 
the  results  of  your  lookout  assessment. 

•  The  assessor  should  restate  the  assessment  objective(s)  and  identify  those  that  you 
successfully  demonstrated. 

•  You  and  the  assessor  should  discuss  the  areas  in  which  you  need  improvement,  and  then 
develop  an  improvement  plan  based  on  the  lookout  assessment  outcome(s). 
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Special  Instructions  for  Lookout  Assessments 

•  You  must  have  vision  correctable  to  at  least  20/40  in  each  eye  and  uncorrected  vision  of 
at  least  20/200  in  each  eye.  Your  color  sense  must  be  determined  to  be  satisfactory, 
without  the  use  of  color  sensing  lenses  (46  CFR  10.205). 

•  AH  phases  of  the  lookout  assessment  are  to  be  conducted  in  the  English  language. 
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ASSESSMENT  CONTROL  SHEET 


Section  1.  Assessment  Reference  Information 


1 .  Name  of  Candidate 

2.  Name  of  Designated  Assessor 

3.  Signature  of  Qualified  Instructor1 

4.  Date  of  Assessment 

5.  Assessment  Location2 

Section  2.  STCW  Reference  Information 


Assessment  Area 

Performance  of  lookout  duties. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

STCW  Requirement 

1995  STCW  Code,  Section  A-ll/4,  p.  68-  Mandatory  minimum  requirements  for 
certification  of  ratings  forming  part  of  a  navigational  watch. 

STCW  Function 

Navigation  at  the  support  level. 

STCW  Competence 

Keep  a  proper  lookout  by  sight  and  hearing. 

STCW  Proficiency 

Responsibilities  of  a  lookout,  including  reporting  the  approximate  bearing  of  a  sound 
signal,  light  or  other  object  in  degrees  or  points. 

Section  3.  Assessment  Objectives,  Types  and  Outcomes 


Assessment  Objective 

Assessment  Type 

Assessment  Outcome 

1 .  Describe  lookout  duties  and  responsibilities. 

Oral  or  written  questions 

□ 

Pass 

□ 

Fail 

2.  Identify  lookout  stations  and  safe  routes. 

Oral  or  written  questions 

□ 

Pass 

□ 

Fail 

3.  Describe  and  identify  the  international  distress 
signals. 

Oral  or  written  questions 

□ 

Pass 

□ 

Fail 

4.  Demonstrate  lookout  techniques  and  make 
lookout  reports  in  clear  visibility  during  daylight. 

Practical  demonstration 

□ 

Pass 

□ 

Fail 

5.  Demonstrate  lookout  techniques  and  make  reports 
in  clear  visibility  at  night. 

Practical  demonstration 

a 

Pass 

□ 

Fail 

6.  Demonstrate  lookout  techniques  and  make 
lookout  reports  in  restricted  visibility  during 
daylight  or  at  night. 

Practical  demonstration 

□ 

Pass 

□ 

Fail 

7.  Demonstrate  the  use  of  lookout  equipment. 

Practical  demonstration 

□ 

Pass 

□ 

Fail 

8.  Demonstrate  man  overboard  procedures. 

Practical  demonstration 

□ 

Pass 

□ 

Fail 

9.  Demonstrate  lookout  watch  relief  procedures. 

Demonstration 

□ 

Pass 

□ 

Fail 

COMMENTS: 

FINAL  SCORE 

□ 

Pass 

□ 

Fail 

1  The  qualified  instructor  certifies  that  the  candidate  has  met  the  training  prerequisites. 

2  Write  the  name  of  the  simulator  or  vessel  used  and  her  location  during  the  assessment  (e.g., 
Massachusetts  Maritime  Academy  training  simulator;  S/R  Long  Beach  -  Day  1  of  Puget  Sound 
to  Valdez  run,  etc.). 
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Section  4.  Assessment  Worksheet  Summary 


Assessment 

Worksheet 

Assessment  Objective 

Action 

1 

1 .  Describe  lookout  duties  and 
responsibilities. 

2.  Identify  lookout  stations  and 
safe  routes  on  board. 

3.  Describe  and  identify  the 
international  distress 
signals. 

1 .1  Describe  lookout  duties  and  responsibilities. 

2.1  Identify  lookout  stations  on  board. 

2.2  Identify  safe  routes  to  and  from  lookout  stations. 

3.1  Describe  and  identify  the  international  distress  signals. 

II 

4.  Demonstrate  lookout 
techniques  and  make 
lookout  reports  in  clear 
visibility  during  daylight 

4.1  List  sightings  that  should  be  reported  when  detected. 

4.2  Report  sighted  objects  using  the  ship’s  bell. 

4.3  Report  sighted  objects  verbally  using  degrees. 

4.4  Report  sighted  objects  verbally  using  points. 

4.5  Report  sky  objects  verbally  using  points. 

4.6  Report  audible  targets  verbally  using  points. 

in 

5.  Demonstrate  lookout 
techniques  and  make 
lookout  reports  in  clear 
visibility  at  night. 

5.1  Report  sighted  objects  using  the  ship’s  bell. 

5.2  Report  sighted  objects  verbally  using  degrees. 

5.3  Report  sighted  objects  verbally  using  points. 

5.4  Report  sky  objects  verbally  using  points. 

5.5  Report  audible  targets  verbally  using  points. 

IV 

6.  Demonstrate  lookout 
techniques  and  make 
lookout  reports  in  restricted 
visibility  during  daylight  or 
at  night. 

6.1  Report  sighted  objects  using  the  ship’s  bell. 

6.2  Report  sighted  objects  verbally  using  degrees. 

6.3  Report  sighted  objects  verbally  using  points. 

6.4  Report  sky  objects  verbally  using  points. 

6.5  Report  audible  targets  verbally  using  points. 

V 

7.  Demonstrate  the  use  of 
lookout  equipment. 

7.1  Demonstrate  the  use  of  ship’s  bell. 

7.2  Demonstrate  the  use  of  internal  communications. 

7.3  Demonstrate  the  use  and  care  of  binoculars. 

7.4  Demonstrate  the  ability  to  take  bearing  using  a  bearing 
repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or 
pelorus. 

7.5  List  personal  equipment  for  lookout. 

VI 

8.  Demonstrate  man 
overboard  procedures. 

8.1  Throw  a  ring  life  buoy  to  the  man  overboard. 

8.2  Sound  the  alarm. 

8.3  Keep  the  victim  in  sight  at  all  times. 

VII 

9.  Demonstrate  lookout  watch 
relief  procedures. 

9.1  Remain  at  your  lookout  station  until  you  are  properly 
relieved. 

9.2  Do  not  allow  anyone  to  relieve  you  until  he  or  she  has 
established  night  vision,  or  if  for  any  reason  you  suspect 
that  your  relief  is  unabfe  to  carry  out  the  duties  of 
lookout. 

9.3  Pass  on  information  to  your  relief. 

9.4  Call  the  bridge  if  you  are  not  properly  relieved. 
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Assessment  Procedures/Lookout  Duties 


ASSESSMENT  WORKSHEET  I 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

1 .  Describe  lookout  duties  and  responsibilities. 

2.  Identify  lookout  stations  and  safe  routes  onboard. 

3.  Describe  and  identify  the  international  distress  signals. 

Assessment  Method 

Oral  or  written  questions. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

Examination  proctor,  test  instrument,  answers,  and  associated  references. 

Initial  Condition 

This  assessment  may  be  conducted  either  before  or  after  the  practical 
demonstration  of  lookout  skills. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance 

Measure 

Performance  Standard 

Score 

1.1  Describe  lookout 
duties  and 
responsibilities. 

Answers  question: 

What  are  the  duties 
and  responsibilities  of 
the  lookout? 

Specifies  that  a  lookout’s  duties  are: 

sighting,  identifying,  and  accurately  reporting 
to  the  responsible  authority  all  objects  or 
sounds  detected. 

□  Pass 

□  Fail 

2.1  Identify  lookout 
stations  on  board. 

Answers  question: 

Identify  three 
common  lookout 
stations  on  this 
vessel. 

Specifies: 

□  Bridge. 

□  Bridge  wings. 

□  Bow. 

□  Pass 

□  Fail 

2.2  Identify  safe  routes  to 
and  from  lookout 
stations. 

Answers  question: 

Identify  the  safe 
routes  to  the  bow 
lookout  station  on  this 
vessel. 

Identifies  safe  routes  to  and  from  bow  lookout 
station. 

□  Pass 

□  Fail 

3.1  Identify  and  describe 
the  international 
distress  signals. 

Answers  question: 

Describe  and  identify 
six  international 
distress  signals. 

Identifies  and  describes  six  of  the  following: 

□  Red  star  shells. 

□  Fog  horn  continuing  sounding. 

□  Flames  on  a  vessel. 

□  Gun  fired  at  intervals  of  1  minute. 

□  Orange  background  black  ball  &  square. 

□  SOS. 

□  “Mayday"  by  radio. 

□  Parachute  red  flare. 

□  Dye  marker  (any  color). 

□  Code  flags  November  Charlie. 

□  Square  flag  and  ball. 

□  Wave  arms. 

O  Radiotelephone  alarm. 

□  Position  indicating  radio  beacon. 

□  Smoke. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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Assessment  Procedures/Lookout  Duties 


ASSESSMENT  WORKSHEET  II 


Section  1.  Assessment  Conditions 


Assessment  Objective 

4.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  during 
daylight. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and 
bearing  repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The  lookout 
station  should  be  clear  and  the  assessor  must  be  able  to  observe  activities. 

Initial  Conditions 

The  assessment  should  be  conducted  in  clear  visibility  during  daylight.  The  assessor 
should  ensure  there  are  reportable  objects  in  sight. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance 

Measure 

Performance  Standard 

Score 

4.1  List  the  sightings 
that  should  be 
reported  when 
detected. 

Answers 

question: 

Identify  six  of 
the  sightings 
that  should  be 
reported  when 
detected  by  the 
lookout. 

Correct  response  includes  six  of  the  following  sightings: 

□  Distress  signals. 

□  Persons  in  distress  (man  overboard,  shipwrecked 
survivors,  calls  for  help). 

□  Vessels  and/or  aircraft  in  distress. 

□  Derelicts,  wreckage,  floating  or  partially  submerged 
debris. 

□  Pollution  incident  (oil  spill,  a  sheen  on  the  water). 

□  Vessel  traffic  and/or  aircraft. 

□  Aids  to  navigation  (navigation  buoys  and  lights). 

□  Danger  to  the  vessel  or  hazards  to  navigation  (land, 
sighting  or  hearing  breaking  surf,  obstructions, 
discoloration  of  the  water). 

□  Reduction  in  visibility  due  to  fog,  mist,  falling,  snow, 
heavy  rainstorms,  sandstorms,  or  any  other  similar 
cause. 

□  Change  in  weather. 

□  Hearing  other  vessels  or  aids  to  navigation  (bell, 
gong,  fog  horn,  or  maneuvering  signals). 

□  Ice. 

□  Unusual  sightings  or  any  unreported  change  in  the 
field  of  view  or  sector. 

□  Sightings  of  marine  life  in  compliance  with  the 
Endangered  Species  Act  /  Marine  Mammal 

Protection  Act  (50  CFR  222),  such  as  humpback  or 
North  Atlantic  right  whales,  stellar  sea  lions,  or  sea 
turtles. 

□  Pass 

□  Fail 

4.2  Report  sighted 
objects  using  the 
ship's  bell. 

Reports  sighted 
objects  using 
the  ship’s  bell 

Reports  at  least  3  surface  objects  using  the  ship’s  bell. 

Improper  reporting  constitutes  failure.  Failure  to  detect 
visible  objects  within  in  a  specified  sector  of  view 
constitutes  failure. 

□  Pass 

□  Fail 
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Assessment  Procedures/Lookout  Duties 


Section  2 -Actions,  Performance  Measures,  Standards,  and  Scores 
(continued) 


4.3  Report  sighted 
objects  verbally 
using  degrees. 

Reports  sighted 
objects  verbally 
using  degrees 

□  Verbally  reports  at  least  3  surface  objects. 

Reports  all  visible  objects. 

□  What  (type)  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports  must 
be  within  22-1/2°  of  the  actual  bearing  of  detected 
targets 

□  How  far-off  (hull-down,  on  the  horizon,  hull-up, 
close  aboard). 

Failure  of  any  of  the  above  four  standards  constitutes 

failure. 

□  Pass 

□  Fail 

4.4  Report  sighted 
objects  verbally 
using  points. 

Reports  sighted 
objects  verbally 
using  points 

□  Verbally  reports  at  least  3  surface  objects. 

Reports  all  visible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports  must 
be  within  ±  2  points  of  the  actual  bearing  of 
detected  targets. 

□  How  far-off  (hull-down,  on  the  horizon,  hull-up, 
close  aboard). 

Failure  of  any  of  the  above  four  standards  constitutes 

failure. 

□  Pass 

□  Fail 

4.5  Report  sky 
objects  verbally 
using  points.3 

Reports  sighted 
objects  verbally 
using  points 

□  Verbally  reports  at  least  3  sky  objects.  Reports  all 
visible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports  must 
be  within  +  2  points  of  the  actual  bearing  of 
detected  targets. 

Failure  of  any  of  the  above  three  standards  constitutes 

failure. 

□  Pass 

□  Fail 

4.6  Report  audible 
target  verbally 
using  points. 

Reports  audible 
target  verbally 
using  points 

□  Verbally  reports  at  least  3  audible  targets. 

Reports  all  audible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true). 

□  Where  (bearings,  relative  or  true).  Reports  must 
be  within  ±  2  points  of  the  actual  bearing  of 
detected  targets. 

Failure  of  any  of  the  above  three  standards  constitutes 
failure. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 

3  Successful  completion  of  this  action  in  any  condition  of  visibility  satisfies  requirement. 
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Assessment  Procedures/Lookout  Duties 


ASSESSMENT  WORKSHEET  III 


Section  1.  Assessment  Conditions 


Assessment  Objective 

5.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  clear  visibility  at  night. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions  and  standards. 

Required  Equipment, 
Apparatus,  and/or 

Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and  bearing 
repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The  lookout  station 
should  be  clear  and  the  assessor  must  be  able  to  observe  activities. 

Initial  Conditions 

The  assessment  should  be  conducted  in  clear  visibility  at  night.  The  assessor  should 
ensure  there  are  reportable  objects  in  sight. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance 

Measure 

Performance  Standard 

Score 

5.1  Report 
sighted 
objects  using 
the  ship’s  bell. 

Reports 
sighted 
objects  using 
the  ship’s  bell 

Reports  at  least  3  surface  objects  using  the  ship's  bell. 

Failure  to  detect  visible  objects  within  in  a  specified  sector  of 
view  constitutes  failure. 

□  Pass 

□  Fail 

5.2  Report 
sighted 
objects 
verbally  using 
degrees. 

Reports 
sighted 
objects 
verbally  using 
degrees 

□  Verbally  reports  at  least  3  surface  objects.  Reports  all 
visible  objects 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  22-1/2°  of  the  actual  bearing  of  detected  targets. 

□  How  far-off  (hull-down,  on  the  horizon,  hull-up,  close 
aboard). 

Failure  of  any  of  the  above  four  standards  constitutes  failure. 

□  Pass 

□  Fail 

5.3  Report 
sighted 
objects 
verbally  using 
points. 

Reports 
sighted 
objects 
verbally  using 
points 

□  Verbally  reports  at  least  3  surface  objects.  Reports  ail 
visible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  ±  2  points  of  the  actual  bearing  of  detected 
targets. 

□  How  far  (hull-down,  on  the  horizon,  hull-up,  close 
aboard). 

Failure  of  any  of  the  above  four  standards  constitutes  failure. 

□  Pass 

□  Fail 
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Assessment  Procedures/Lookout  Duties 


Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


5.4  Report  sky 
objects 
verbally  using 
points. 

Reports 
sighted 
objects 
verbally  using 
points 

□  Verbally  reports  at  least  3  surface  objects.  Reports  all 
visible  objects 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  +  2  points  of  the  actual  bearing  of  detected 
targets. 

□  How  far  (hull-down,  on  the  horizon,  hull-up,  close 
aboard). 

Failure  of  any  of  the  above  three  standards  constitutes  failure.. 

□  Pass 

□  Fail 

5.5  Report 

audible  target 
verbally  using 
points. 

Reports 
audible  target 
verbally  using 
points 

□  Verbally  reports  at  least  3  audible  targets.  Reports  all 
audible  objects. 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  ±  2  points  of  the  actual  bearing  of  detected 
targets. 

Failure  of  any  of  the  above  three  standards  constitute  failure. 

□  Pass 

□  Fail 

j 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 

4  Successful  completion  of  this  action  in  any  condition  of  visibility  satisfies  requirement. 
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ASSESSMENT  WORKSHEET  IV 


Section  1.  Assessment  Conditions 


Assessment  Objective 

6.  Demonstrate  lookout  techniques  and  make  lookout  reports  in  restricted  visibility. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  tocus  binoculars,  and 
bearing  repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The  lookout 
station  should  be  clear  and  the  assessor  must  be  able  to  observe  activities. 

Initial  Conditions 

The  assessment  should  be  conducted  in  restricted  visibility  during  daylight  or  at  night. 
The  assessor  should  ensure  there  are  reportable  objects  in  sight. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance 

Measure 

Performance  Standard 

Score 

6.1  Report 
sighted 
objects  using 
the  ship’s  bell. 

Reports 
sighted  objects 
using  the 
ship’s  bell 

Reports  at  least  3  surface  objects  using  the  ship’s  bell. 

Failure  to  detect  visible  objects  within  in  a  specified  sector  of 
view  constitutes  failure. 

□  Pass 

□  Fail 

6.2  Report  sighted 
objects 
verbally  using 
degrees. 

Reports 
sighted  objects 
verbally  using 
degrees 

□  Verbally  reports  at  least  3  surface  objects.  Reports  all 
visible  objects 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  22-1/2°  of  the  actual  bearing  of  detected  targets. 

□  How  far  (hull-down,  on  the  horizon,  hull-up,  close 
aboard). 

Failure  of  any  of  the  above  four  standards  constitute  failure. 

□  Pass 

□  Fail 

6.3  Report  sighted 
objects 
verbally  using 
points. 

Reports 
sighted  objects 
verbally  using 
points 

□  Verbally  reports  at  least  3  surface  objects.  Reports  all 
visible  objects. 

□  What  (type  of  objects). 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  ±  2  points  of  the  actual  bearing  of  detected  targets. 

□  How  far  (hull-down,  on  the  horizon,  hull-up,  close 
aboard). 

Failure  of  any  of  the  above  four  standards  constitutes  failure. 

□  Pass 

□  Fail 

6.4  Report  sky 
objects 
verbally  using 
points.*” 

Reports 
sighted  objects 
verbally  using 
points 

□  Verbally  reports  at  least  3  surface  objects.  Reports  all 
visible  objects. 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  ±  2  points  of  the  actual  bearing  of  detected 
targets. 

Failure  of  any  of  the  above  three  standards  constitutes  failure. 

□  Pass 

□  Fail 

5  Successful  completion  of  this  action  in  any  condition  of  visibility  satisfies  requirement. 
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Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


6.5  Report  audible 

Reports 

□  Verbally  reports  at  least  3  audible  targets.  Reports  all 

□  Pass 

targets  verbally 

audible  targets 

audible  objects. 

□  Fail 

using  points.6 

verbally  using 
points 

□  What  (type  of  objects) 

□  Where  (bearings,  relative  or  true).  Reports  must  be 
within  +  2  points  of  the  actual  bearing  of  detected 
targets. 

Failure  of  any  of  the  above  three  standards  constitutes  failure. 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 


6  Successful  completion  of  this  action  in  any  condition  of  visibility  satisfies  requirement. 
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Assessment  Procedures/Lookout  Duties 


ASSESSMENT  WORKSHEET  V 


Section  1.  Assessment  Conditions 


Assessment  Objective 

7.  Demonstrate  the  use  of  lookout  equipment. 

Assessment  Method 

Practical  skill  demonstration,  shipboard  assessment  and/or  ship  bridge  simulator 
assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and 
standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and 
bearing  repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The 
lookout  station  should  be  clear  and  the  assessor  must  be  able  to  observe 
activities. 

Initial  Conditions 

The  assessment  may  be  conducted  in  any  condition  of  visibility.  It  is  essential 
that  reportable  objects  be  in  sight. 

Section  2.  Performance  Measures,  Standards,  and  Scores 


Action 

Performance 

Measure 

Performance  Standard 

Score 

7.1  Demonstrate  the  use  of 
ship’s  bell. 

Reports  sighted 
objects  using  the 
ship’s  bell. 

Satisfied  by  item  3.2  (Reporting  sighted 
objects  using  the  ship’s  bell). 

□  Pass 

□  Fail 

7.2  Demonstrate  the  use  of 
internal  communications 
(sound-powered 
telephone,  walkie-talkie, 
etc.). 

Reports  sighted 
objects  verbally. 

Satisfied  by  items  3.3  (Reporting  sighted 
objects  verbally  using  degrees  on  the 
sound-powered  telephone);  3.4  (Reporting 
sighted  objects  verbally  using  points  on 
the  sound-powered  telephone);  and  3.5 
(Reporting  audible  target  verbally  using 
points  on  the  sound-powered  telephone). 

□  Pass 

□  Fail 

7.3  Demonstrate  the  use, 
care  and  focusing  of 
binoculars. 

Detects  visual  targets 
using  binoculars. 

Describes  care  and 
use  of  binoculars. 

□  Demonstrates  focus  by  identifying  far 
object  requiring  use  of  binoculars. 

□  Demonstrates  systematic  scanning 
technique. 

□  Identifies  correct  stowage 
procedures. 

□  Pass 

□  Fail 

7.4  Demonstrate  the  ability 
to  take  bearing  using  a 
bearing  repeater  fitted 
with  a  bearing/azimuth 
circle,  alidade,  or 
pelorus. 

Observes  bearing 
using  a  bearing 
repeater  fitted  with  a 
bearing/azimuth  circle, 
alidade,  or  pelorus. 

Sighted  objects  are  observed  within  ±  3° 
of  the  actual  bearing. 

■ 

□  Pass 

□  Fail 

7.5  List  personal  equipment 
for  lookout  (flash  light, 
sun  glasses,  foul 
weather  gear,  etc.). 

Answers  question: 

Identify  the  personal 
equipment  you  should 
bring  with  you  when 
you  perform  lookout. 

At  a  minimum,  correct  response  includes: 

□  Flashlight. 

□  Sunglasses. 

□  Foul  weather  gear. 

□  Hat. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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ASSESSMENT  WORKSHEET  VI 


Section  1 .  Assessment  Conditions 


Assessment  Objective 

8.  Demonstrate  man  overboard  procedures. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and 
standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and 
bearing  repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The 
lookout  station  should  be  clear  and  the  assessor  must  be  able  to  observe 
activities. 

Initial  Conditions 

The  assessment  may  be  conducted  in  any  condition  of  visibility.  It  is  essential 
that  reportable  objects  be  in  sight.  The  designated  assessor  simulates  man 
overboard  to  starboard  or  port. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

8.1  Throw  a  ring  life  buoy  to 
the  man  overboard. 

Throws  ring  life  buoy. 

□  Releases  ring  life  buoy  immediately. 

□  Throws  it  over  the  person  in  the  water 
so  that  he/she  can  grab  the  floating 
line  first. 

□  Exercises  caution  not  to  strike  the 
person  with  the  ring  life  buoy. 

□  Pass 

□  Fail 

8.2  Sound  the  alarm. 

Reports  man  overboard 
to  the  bridge. 

□  Immediately  shouts  “Man  overboard- 
Man,  overboard-Man  overboard  to 
starboard  (or  port).” 

□  Indicates  the  side  on  which  the 
person  fell  over  in  order  to  assist  the 
bridge  watch  in  determining  the 
proper  avoidance  action. 

□  Pass 

□  Fail 

8.3  Keep  the  victim  in  sight 
at  all  times. 

Points  at  the  victim. 

□  Establishes  and  maintains  visual 
contact. 

□  Points  in  the  direction  of  the  person  in 
the  water. 

□  Remains  on  station  until  he  or  she  is 
properly  relieved  or  ordered 
otherwise. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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ASSESSMENT  WORKSHEET  VII 


Section  1.  Assessment  Conditions 


Assessment  Objective 

9.  Demonstrate  the  lookout  watch  relief  procedure. 

Assessment  Method 

Practical  skill  demonstration,  shipboard  assessment  and/or  ship  bridge  simulator 
assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and 
standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  a  lookout  station  equipped  with  an  internal 
communications  system,  ship’s  bell,  7x50  individual  eye  focus  binoculars,  and 
bearing  repeater  fitted  with  a  bearing/azimuth  circle,  alidade,  or  pelorus.  The 
lookout  station  should  be  clear  and  the  assessor  must  be  able  to  observe 
activities. 

Initial  Conditions 

The  assessment  may  be  conducted  in  any  condition  of  visibility.  It  is  essential 
that  reportable  objects  are  in  sight. 

Section  2.  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

9.1  Remain  at  your  lookout 
station  until  you  are 
properly  relieved. 

Answers  question: 

When  can  a  lookout 
leave  his  or  her  station 
before  being  relieved? 

Correct  response  is  never. 

□  Pass 

□  Fail 

9.2  Do  not  allow  anyone  to 
relieve  you  until  they 
have  established  their 
night  vision  or  if  for  any 
reason,  you  suspect  that 
your  relief  is  unable  to 
carry  out  the  duties  of 
lookout. 

Answers  question: 

When  being  relieved 
during  darkness ,  what  is 
an  important 
consideration  before 
handing  over  the  watch 
to  your  relief? 

Correct  response  is  that  the  lookout 
should  ensure  that  his  or  her  relief  has 
established  night  vision  before  handing 
over  the  watch. 

□  Pass 

□  Fail 

9.3  Pass  on  information  to 
your  relief. 

Answers  question: 

What  items  of 
information  constitute 
an  adequate  watch 
relief  report? 

Correct  response  includes  all  of  these 
items: 

□  The  location  of  any  object(s)  in  sight. 

□  Object(s)  previously  reported. 

□  The  present  and  past  weather. 

□  Any  special  instructions  or  safety 
precautions. 

□  Brief  description  of  your  watch 
activities. 

□  The  status  of  the  navigation  lights. 

□  Pass 

□  Fail 

9.4  Call  the  bridge  if  you  are 
not  properly  relieved. 

Answers  question: 

What  action  do  you  take 
if  you  are  not  properly 
relieved? 

Correct  response  is  to  report  to  bridge  that 
no  relief  has  arrived. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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Appendix  B  to 

“A  Method  for  Developing  Mariner  Assessments” 


MARINER  ASSESSMENT  PROCEDURES  FOR  THE 
PERFORMANCE  OF  HELMSMAN  DUTIES 


This  appendix  contains  example  procedures  for  assessing  a 
mariner’s  ability  to  perform  helmsman  duties.  The  assessment 
package  consists  of  the  following:  assessor  instructions,  candidate 
instructions,  five  assessment  worksheets,  and  an  assessment 
control  sheet  for  documenting  the  final  results  of  the  assessment. 


This  report  may  be  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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ASSESSOR  INSTRUCTIONS 


Introduction 

The  following  procedures  are  designed  for  assessing  a  candidate’s  ability  to  perform  helmsman 
duties.  The  assessment  includes  demonstrating  the  use  of  all  steering  and  compass  equipment; 
performing  steering  procedures  in  open  waters;  and  performing  steering  procedures  in  shallow 
water  while  maneuvering.  It  also  includes  demonstration  of  steering  by  visual  range  or 
landmark. 

The  assessment  materials  include  Assessor  Instructions,  Candidate  Instructions,  an  Assessment 
Control  Sheet,  and  five  separate  assessment  modules  ( Assessment  Worksheets  I-V).  The 
Assessor  Instructions  describe  the  assessment  objectives,  method,  performance  measures,  and 
performance  standards.  The  Assessor  Instructions  also  list  the  responsibilities  of  the  assessor 
during  each  phase  of  the  assessment  process.  The  Candidate  Instructions  explain  the  assessment 
process  from  the  candidate’s  perspective  and  define  the  responsibilities  of  the  candidate. 

The  Assessment  Control  Sheet  provides  a  record  of  the  names  of  the  candidate,  assessor,  and 
qualified  instructor  who  certified  that  the  candidate  met  the  training  prerequisites  for  assessment. 
The  control  sheet  also  includes  the  date  and  location  of  the  assessment  and  the  STCW  reference 
information  for  the  competency  area  being  assessed.  Section  3  of  the  Assessment  Control  Sheet 
provides  space  for  the  assessor  to  record  the  candidate’s  score  on  each  assessment  objective. 
Finally,  section  4  of  the  control  sheet  summarizes  the  assessment  objectives  and  the  actions 
comprising  each  objective. 

As  the  assessor,  you  will  evaluate  the  candidate’s  ability  to  complete  the  assessment  objectives 
correctly.  Using  the  scoring  procedures  described  below,  you  will  provide  the  candidate  with  a 
separate  score  for  each  assessment  objective  and  compile  these  scores  into  a  final  score  for  the 
entire  assessment.  The  scores  for  each  assessment  objective  and  the  candidate’s  overall  score  for 
the  entire  assessment  are  to  be  recorded  on  the  Assessment  Control  Sheet. 

The  rest  of  this  section  briefly  describes  the  various  components  of  this  assessment.  Assessment 
Worksheets  I  through  V provide  additional  information  about  the  assessment  objectives,  method, 
conditions,  performance  measures,  and  performance  standards. 

Assessment  Objectives 

During  this  assessment,  the  candidate  should  demonstrate  the  following  abilities: 

1 .  Use  steering  equipment. 

2.  Use  magnetic  and  gyrocompasses  and  visual  range. 

3.  Understand  and  execute  helm  orders  in  open  waters  at  sea  speed. 

4.  Perform  steering  procedures  in  open  waters  at  sea  speed. 

5.  Understand  and  execute  helm  orders  in  shallow  water  while  maneuvering. 
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6.  Perform  steering  procedures  in  shallow  water  while  maneuvering. 

7.  Understand  and  execute  helm  orders  while  steering  by  visual  range  or  landmark. 

8.  Perform  steering  procedures  while  steering  by  visual  range  or  landmark. 

9.  Perform  steering  duty  changeover. 

1 0.  Perform  helm  relief  procedures. 

1 1 .  Perform  helmsman  steering  failure  procedures  in  open  waters  at  reduced  speed. 

Assessment  Method 

Although  candidates  can  demonstrate  their  knowledge  of  the  helmsman  duties  in  a  written  test, 
to  ensure  proficiency  in  the  actual  operating  environment,  they  should  undergo  assessment 
requiring  practical  demonstration  either  in  a  bridge  simulator  or  onboard  ship.  This  assessment 
package  can  be  applied  using  either  method 

Assessment  Conditions 

The  assessment  should  occur  at  an  approved  steering  station  that  is  fitted  with  manual  and 
automatic  helm  modes,  gyro  repeater,  and  magnetic  compass.  Optimally,  the  steering  system 
should  have  manual  and  automatic  pilot,  non-follow-up  unit,  navigation  modes  with  adjustable 
controls,  rate  of  turn  indicator,  and  steering  alarms.  As  noted  above,  four  different  conditions  are 
required  for  this  assessment:  open  waters  at  sea  speed,  shallow  waters  while  maneuvering,  open 
waters  at  reduced  speed,  and  steering  by  visual  range  or  landmark. 

Performance  Measures  and  Standards 

Each  assessment  objective  is  comprised  of  one  or  more  actions.  Each  action  has  one  or  more 
corresponding  performance  measures  and  standards.  For  example,  assessment  objective  10 
requires  two  different  actions  (see  section  2  of  Assessment  Worksheet  IV  on  p.  15).  Note  that 
within  one  action  there  may  be  more  than  one  performance  measure.  For  example,  in  the  case  of 
action  10.1,  the  candidate  has  to  pass  both  performance  measures  in  order  to  pass  the  action.  If  a 
candidate  fails  one  measure,  he  or  she  fails  the  action. 

Note  that,  if  a  candidate  fails  an  assessment  objective,  he  or  she  automatically  fails  the  entire 
assessment.  If  this  occurs,  you  should  terminate  the  assessment. 

Assessment  Checklist 

The  following  checklist  summarizes  the  tasks  and  responsibilities  of  the  assessor  at  each  phase  of 
the  assessment  process. 

Preparing  for  the  Steering  Assessment 

□  Gather  steering  assessment  materials. 

□  Prepare  the  assessment  area(s),  test  facility,  and/or  bridge. 
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□  Prepare  and  arrange  for  necessary  steering  equipment,  and  ensure  that  all  equipment  is 
operational. 

a  Review  the  Assessment  Control  Sheet,  Assessment  Worksheets,  and  Candidate  Instructions. 

□  If  you  are  using  a  simulator,  check  all  exercise  scenarios  to  ensure  they  include  the  required 
assessment  conditions: 

•  Open  waters  at  sea  speed  {Assessment  Worksheet  I). 

•  Shallow  water  while  maneuvering  {Assessment  Worksheet  II). 

•  Steering  by  visual  range  or  landmark  (Assessment  Worksheet  III). 

•  Any  of  the  above  {Assessment  Worksheet  IV). 

•  Open  waters  at  reduced  speed  (Assessment  Worksheet  V). 

□  If  you  are  conducting  a  shipboard  assessment,  ensure  that  the  appropriate  conditions  are 
present  for  each  Assessment  Worksheet  and  that  the  vessel  is  clear  of  navigational  hazards.  A 
steering  failure  drill  may  be  simulated  to  fulfill  the  conditions  required  for  Assessment 
Worksheet  V. 

□  Identify  and  observe  necessary  safety  precautions. 

□  Schedule  assessment  and  inform  all  affected  personnel. 

Briefing  the  Candidate  before  the  Assessment 

□  Provide  the  candidate  with  a  copy  of  the  Candidate  Instructions  and  Assessment  Control 
Sheet. 

□  Review  the  steering  assessment  instructions  with  the  candidate  and  answer  any  questions. 

□  Discuss  the  assessment  objectives  and  explain  any  that  require  the  use  of  non-standard 
procedures. 

□  Explain  the  steering  assessment  performance  measures  and  standards. 

□  Advise  the  candidate  of  the  time  constraints,  including  when  the  assessment  will  begin  and 
under  what  circumstances  the  assessment  will  be  terminated. 

□  Inform  the  candidate  that  he  or  she  should  use  appropriate  marine  terminology  and 
nomenclature  during  all  phases  of  the  assessment. 

□  Discuss  the  desired  outcome(s)  and  consequences  of  failing  to  perform  part  or  all  of  the 
steering  assessment. 

□  Remind  the  candidate  that  it  is  permissible  to  ask  questions  during  the  assessment,  especially 
if  he  or  she  has  a  safety  concern. 

□  Inform  the  candidate  that  all  phases  of  the  assessment  are  to  be  conducted  in  English. 

□  Discuss  the  candidate’s  willingness  to  be  assessed  under  the  circumstances  presented. 
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Observing  the  Candidate’s  Performance 

□  If  a  safety  violation  occurs,  terminate  the  assessment  immediately. 

□  Continuously  observe  the  candidate  during  the  assessment.  Require  that  standard  procedures 
or  company  policy  be  adhered  to,  except  when  assessment  procedures  require  demonstration 
of  knowledge  or  skill  different  from  a  convention  adopted  by  the  company. 

□  Give  the  candidate  at  least  five  helm  commands  during  a  30-minute  period  on  the  helm. 

□  Ensure  realistic  assessment  conditions  consistent  with  a  normal  working  environment  for  the 
helmsman.  Ensure  that  the  candidate  can  concentrate  on  the  task  at  hand.  Do  not  allow  other 
crew  members  to  interfere  with  the  assessment.  Do  not  allow  the  candidate  to  “learn  the 
test”  by  observing  the  performance  of  other  assessment  candidates. 

□  Avoid  giving  the  candidate  unsolicited  assistance,  but  respond  to  appropriate  questions  and 
provide  appropriate  equipment  when  required. 

□  Remain  objective  and  maintain  positive  control  of  the  assessment  process  at  all  times. 

Recording  Results  and  Determining  Assessment  Outcome 

□  Record  performance  on  the  appropriate  steering  assessment  worksheet. 

□  The  steering  standards  specified  on  each  Assessment  Worksheet  may  be  modified  to 
compensate  for  the  area  of  navigation  and  prevailing  sea  state,  but  otherwise  strictly  adhere 
to  the  prescribed  scoring  procedure(s). 

□  Determine  and  document  the  outcome  of  the  assessment,  then  transfer  the  final  results  to  the 
Assessment  Control  Sheet. 

□  Attest  to  successful  demonstration  of  tasks  in  the  Training  Record  Book  (TRB),  if 
appropriate. 

Debriefing  the  Candidate 

□  Debrief  the  candidate  as  soon  as  possible  after  the  assessment. 

□  Provide  the  candidate  with  a  copy  of  the  Assessment  Control  Sheet. 

□  Restate  the  assessment  objectives. 

□  Focus  on  positive  accomplishments  first. 

□  Identify  areas  needing  improvement. 

□  If  the  candidate  failed  to  demonstrate  proficiency,  jointly  develop  an  improvement  plan  to 
prepare  for  reassessment. 


Appendix  B  of A  Method  for  Developing  Mariner  Assessments 


Assessment  Procedures/Helmsman  Duties 


CANDIDATE  INSTRUCTIONS 


In  this  assessment,  you  will  be  evaluated  on  your  ability  to  perform  helmsman  duties.  Table  1 
shows  the  1995  STCW  Code  specification  for  the  performance  of  helmsman  duties. 


Table  1.  STCW  Code  specification  for  the  performance  of  helmsman  duties. 


STCW  Requirement 

1995  STCW  Code,  Section  A-ll/4  (p.  68):  Mandatory  minimum  requirements 
for  certification  of  ratings  forming  part  of  a  navigational  watch. 

STCW  Function 

Navigation  at  the  support  level. 

STCW  Competence 

Steer  the  ship  and  comply  with  helm  orders  also  in  the  English  language. 

STCW  Proficiency 

Performance  of  helmsman  duties. 

Assessment  Objectives  and  Methods 

A  qualified  assessor  will  assess  you  on  your  ability  to  meet  the  1 1  assessment  objectives  in  the 
first  column  of  Table  2.  Note  that  each  assessment  objective  has  a  corresponding  assessment 
method(s).  Performance  assessed  by  “practical  skill  demonstration”  means  your  assessor  will 
ask  you  to  demonstrate  your  ability  to  perform  the  actions  required  in  the  objective. 

Performance  assessed  by  “oral  or  written  test  questions”  means  your  assessor  will  ask  you  to 
respond  to  one  or  more  questions.  He  or  she  will  assess  your  ability  to  respond  correctly  to  each 
question.  Your  practical  skill  demonstration  will  occur  either  onboard  ship  while  underway  or  in 
a  ship  bridge  simulator. 


Table  2.  Assessment  objectives  and  methods  for  helmsman  duties. 


Assessment  Objective 

Assessment  Method 

1 .  Demonstrate  use  of  steering  equipment. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge 
simulator,  oral  or  written  test  questions 

2.  Demonstrate  the  use  of  magnetic  and 
gyrocompasses  and  visual  range. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge 
simulator,  oral  or  written  test  questions 

3.  Demonstrate  understanding  and  execution  of 
helm  orders  in  open  waters  at  sea  speed. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

4.  Perform  steering  procedures  in  open  waters  at 
sea  speed. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

5.  Demonstrate  understanding  and  execution  of 
helm  orders  in  shallow  water  while 
maneuvering. 

Practical  skill  demonstration  — 
shipboard  and/or  ship  bridge  simulator 

6.  Perform  steering  procedures  in  shallow  water 
while  maneuvering. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 
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Assessment  Objective 

Assessment  Method 

7.  Demonstrate  understanding  and  execution  of 
helm  orders  while  steering  by  visual  range  or 
landmark. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

8.  Perform  steering  procedures  while  steering  by 
visual  range  or  landmark. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

9.  Demonstrate  steering  duty  changeover. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

10.  Perform  helm  relief  procedures. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge 
simulator,  oral  or  written  test  questions 

1 1 .  Demonstrate  helmsman  steering  failure 
procedures. 

Practical  skill  demonstration  - 
shipboard  and/or  ship  bridge  simulator 

Refer  to  section  4  of  the  Assessment  Control  Sheet  for  a  list  of  the  specific  actions  required  for 
each  objective  (you  should  receive  a  copy  of  this  document  from  the  assessor  during  your  pre¬ 
assessment  briefing).  Below  are  some  general  guidelines  for  what  you  should  expect  during  the 
assessment  process. 

Pre-Assessment  Briefing  with  Assessor 

This  briefing  should  occur  approximately  a  week  in  advance  of  the  scheduled  assessment,  if 
possible.  This  will  help  you  and  the  assessor  to  be  well  prepared  for  the  assessment  when  it 
actually  occurs.  During  this  briefing,  you  should: 

•  Discuss  your  prior  experience,  training,  and/or  company  policy  with  the  assessor.  On  the 
basis  of  these  qualifications,  discuss  whether  you  are  qualified  to  undertake  the 
helmsman  assessment.  If  you  both  agree  you  are  qualified,  then  continue  with  the 
assessment  process.  If  not,  arrange  for  additional  on-the-job  or  simulator  training,  and  set 
a  date  for  another  review  of  your  qualifications. 

•  Obtain  a  copy  of  the  Assessment  Control  Sheet  from  your  assessor.  Review  it  and  discuss 
the  scope  and  depth  of  knowledge  covered  by  the  helmsman  assessment.  Note  that  the 
assessment  is  to  be  conducted  in  the  English  language.  Ask  questions  about  any  part  of 
the  assessment  that  is  unclear. 

•  Review  the  helmsman  assessment  performance  measures  and  standards,  and  ask  any 
questions  you  have  about  them. 

•  Discuss  the  desired  outcome(s)  and  the  consequences  of  failing  to  perform  any  part  of  the 
helmsman  assessment. 

•  Discuss  the  general  helmsman  assessment  procedures.  Your  assessor  will  inform  you  of 
how  much  time  is  allowed,  when  the  assessment  begins,  and  under  what  circumstances 
he  or  she  will  terminate  the  assessment. 
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•  Discuss  any  non-standard  procedures  that  are  expected  of  you  during  the  assessment. 

•  Consider  whether  you  are  willing  to  be  assessed  under  the  circumstances  presented  and 
advise  the  assessor  of  your  willingness  to  undertake  the  assessment. 

Participation  in  the  Assessment 

Your  assessor  will  continuously  observe  you  during  the  assessment.  You  are  expected  to  adhere 
to  standard  procedures  or  company  policy  unless  the  assessor  briefs  you  on  a  requirement  to 
perform  a  non-standard  procedure.  During  the  assessment,  remember  to: 

•  Use  appropriate  marine  terminology  and  nomenclature  at  all  times. 

•  Listen  to  an  entire  question  before  responding  or  acting.  Remember  that  you  may  not  use 
reference  material  of  any  kind  during  the  assessment  examination. 

•  Ask  questions  if  you  have  a  safety  concern.  The  assessor  will  provide  you  with 
appropriate  responses  to  your  questions.  If  a  safety  violation  occurs,  the  assessor  will 
terminate  the  assessment  immediately. 

Your  assessor  will  ensure  realistic  assessment  conditions  consistent  with  the  normal  working 
environment  for  a  helmsman. 

The  Outcome  of  Your  Assessment 

The  assessor  will  record  your  performance  on  a  series  of  Assessment  Worksheets.  The  assessor 
will  score  each  assessment  objective  on  a  “Pass/Fail”  basis.  Acceptable  performance  (a  “Pass” 
score)  will  be  based  on  your  demonstrated  ability  to  correctly  respond  to  the  questions  that  you 
are  asked  during  the  examination  portions  of  the  assessment.  It  will  also  be  based  on  your  ability 
to  safely  perform  assigned  tasks  in  a  manner  that  demonstrates  you  possess  the  required  level  of 
skill,  knowledge,  and  ability,  as  well  as  sound  and  professional  judgment. 

Unacceptable  performance  (a  “Fail”  score)  will  be  based  on  your  failure  to  perform  a  critical 
phase  of  the  assessment  exercise  at  an  acceptable  level  of  proficiency.  When  this  occurs,  your 
assessment  will  be  suspended  and  postponed  until  you  receive  further  instruction  and  training. 

Note  that  the  following  events  will  terminate  an  assessment  immediately: 

•  An  action,  or  lack  of  action,  by  you  which  required  corrective  action  or  intervention  by 
the  assessor  to  prevent  injury,  damage,  or  the  development  of  a  hazardous  condition. 

•  Your  failure  to  use  proper  procedures,  including  appropriate  communication  procedures, 
during  the  assessment. 

•  Your  failure  to  take  prompt  corrective  action  when  required. 

The  assessor  will  strictly  adhere  to  pre-determined  scoring  procedures,  and  will  document  the 
outcome  of  the  assessment.  He  or  she  will  transfer  the  final  results  of  your  assessment  to  the 
Assessment  Control  Sheet. 
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Assessment  Debriefing 

You  and  the  assessor  should  discuss  the  assessment  results  as  soon  as  possible  after  the 
assessment.  During  the  debriefing: 

•  The  assessor  should  provide  you  with  a  copy  of  the  Assessment  Control  Sheet  describing 
the  results  of  your  helmsman  assessment. 

•  The  assessor  should  restate  the  assessment  objective(s)  and  identify  those  that  you 
successfully  demonstrated. 

•  If  appropriate,  you  and  the  assessor  should  discuss  the  areas  in  which  you  need 
improvement,  and  then  develop  an  improvement  plan  based  on  the  helmsman  assessment 
outcome(s). 
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ASSESSMENT  CONTROL  SHEET 


Section  1.  Assessment  Reference  Information 


1 .  Name  of  Candidate 

2.  Name  of  Designated  Assessor 

3.  Signature  of  Qualified  Instructor1 

4.  Date  of  Assessment 

5.  Assessment  Location2 

Section  2.  STCW  Reference  Information 


Assessment  Area 

Performance  of  helmsman  duties. 

Assessment  Method 

Shipboard  and/or  ship  bridge  simulator  assessment. 

STCW  Requirement 

STCW  Code,  Section  A-ll/4,  p.  68  -  Mandatory  minimum  requirements  for 
certification  of  ratings  forming  part  of  a  navigational  watch. 

STCW  Function 

Navigation  at  the  support  level. 

STCW  Competence 

Steer  the  ship  and  comply  with  helm  orders  also  in  the  English  language. 

STCW  Proficiency 

Performance  of  helmsman  duties. 

Section  3.  Assessment  Objectives,  Methods,  and  Scores 


Assessment  Objective 

Assessment  Methods 

Score 

1 .  Demonstrate  use  of  steering  equipment. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator,  oral  test  questions 

□  Pass 

□  Fail 

2.  Demonstrate  the  use  of  magnetic, 
gyrocompasses  and  visual  range. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator,  oral  test  questions 

□  Pass 

□  Fail 

3.  Demonstrate  understanding  and  execution  of 
helm  orders  in  open  waters  at  sea  speed. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

4.  Perform  steering  procedures  in  open  waters 
at  sea  speed. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

5.  Demonstrate  understanding  and  execution  of 
helm  orders  in  shallow  water  while 
maneuverinq . 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

6.  Perform  steering  procedures  in  shallow  water 
while  maneuvering. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

7.  Demonstrate  understanding  and  execution  of 
helm  orders  while  steering  by  visual  range  or 
landmark. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 
a  Fail 

8.  Perform  steering  procedures  while  steering 
by  visual  range  or  landmark. 

Practical  skill  demonstration  —  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

9.  Demonstrate  steering  duty  changeover. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

10.  Perform  helm  relief  procedures. 

Practical  skill  demonstration  — shipboard  and/or 
ship  bridge  simulator,  oral  test  questions 

□  Pass 

□  Fail 

1 1 .  Demonstrate  helmsman  steering  failure 
procedures. 

Practical  skill  demonstration  -  shipboard  and/or 
ship  bridge  simulator 

□  Pass 

□  Fail 

COMMENTS: 

FINAL  SCORE 

□  Pass 

□  Fail 

1  The  qualified  instructor  certifies  that  the  candidate  has  met  the  training  prerequisites. 

2  Write  the  name  of  the  simulator  or  vessel  used  and  her  location  during  the  assessment 
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Section  4.  Assessment  Worksheet  Summary 


Assessment 

Worksheet 

Assessment  Objective 

Action 

1 

1 .  Demonstrate  the  use  of 
steering  equipment. 

1 .1  Use  and  adjustment  of  steering  controls. 

1 .2  Use  of  the  steering  compass. 

1 .3  Use  of  the  rudder  angle  indicator. 

1 .4  Use  of  the  rate  of  turn  indicator. 

1 .5  Use  of  steering  system  alarms. 

1 .6  Use  and  adjustment  of  the  course  recorder. 

2.  Demonstrate  the  use  of 
magnetic  and  gyrocompasses 
and  visual  range. 

2.1  Compare  and  report  course  by  gyro  and 
magnetic  after  course  change. 

2.2  Compare  and  report  course  by  gyro  and 
magnetic  at  time  interval  specified  by 
standing  orders  or  company  policy. 

2.3  Report  course  by  gyro  and  magnetic  to  the 
watch  officer  when  changes  in  compared 
headings  are  noted. 

3.  Demonstrate  understanding 
and  execution  of  helm  orders 
in  open  waters  at  sea  speed 
(water  depth  =  6  times  draft  or 
more). 

3.1  Repeat  helm  command  as  given. 

3.2  Execute  course  change. 

3.3  Execute  helm  order. 

3.4  Report  helm  order. 

3.5  Report  conflicts  in  commands. 

4.  Perform  steering  procedures  in 
open  waters  at  sea  speed. 

4.1  Steer  by  gyrocompass. 

4.2  Steer  by  magnetic  compass. 

4.3  Steer  by  visual  range  or  landmark. 

II 

5.  Demonstrate  understanding 
and  execution  of  helm  orders 
in  shallow  water  while 
maneuvering  (water  depth  = 

1.5  times  draft  or  less). 

5.1  Repeat  helm  command  as  given. 

5.2  Execute  course  change. 

5.3  Execute  helm  order. 

5.4  Report  helm  order. 

5.5  Report  command  conflicts. 

6.  Perform  steering  procedures  in 
shallow  water  while 
maneuvering. 

6.1  Steer  by  gyrocompass. 

6.2  Steer  by  magnetic  compass. 

6.3  Steer  by  visual  range  or  landmark. 
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Section  4.  Assessment  Worksheet  Summary  (continued) 


III 

7.  Demonstrate  understanding 
and  execution  of  helm 
orders  while  steering  by 
visual  range  or  landmark. 

8.  Perform  steering 
procedures  while  steering 
by  visual  range  or 
landmark. 

7.1  Repeat  helm  command  as  given. 

7.2  Execute  course  change. 

7.3  Execute  helm  order. 

7.4  Report  helm  order. 

.5  Report  command  conflicts. 

8.1  Steer  by  gyrocompass. 

8.2  Steer  by  magnetic  compass. 

8.3  Steer  by  visual  range  or  landmark. 

IV 

9.  Demonstrate  steering  duty 
changeover. 

9.1  Changeover  steering  pumps. 

9.2  Changeover  steering  motors. 

9.3  Engage/disengage  autopilot. 

9.4  Engage/disengage  non-follow-up  unit  (NFU). 

10.  Perform  helm  relief 
procedures. 

10.1  Hand  over  the  helm. 

10.2  Relieving  the  wheel  during  a  maneuver. 

V 

1 1 .  Demonstrate  helmsman 
steering  failure  procedures. 

1 1 .1  Respond  to  steering  failure. 
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ASSESSMENT  WORKSHEET  I 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

1 .  Demonstrate  the  use  of  steering  equipment. 

2.  Demonstrate  the  use  of  magnetic  and  gyrocompasses  and  visual  range. 

3.  Demonstrate  understanding  and  execution  of  helm  orders  in  open  waters  at 
sea  speed  (water  depth  =  6  times  draft  or  more). 

4.  Perform  steering  procedures  in  open  waters  at  sea  speed. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  an  approved  steering  station  fitted  with  manual 
and  automatic  helm  modes,  gyro  repeater,  and  magnetic  compass.  Optimally,  the 
steering  system  should  have  manual  and  automatic  pilot,  non-follow-up  unit,  and 
navigation  modes  with  adjustable  controls,  rate  of  turn  indicator,  and  steering 
alarms. 

Initial  Condition 

This  assessment  should  occur  in  open  waters  at  sea  speed.  Water  depth  must  be 
at  least  six  times  vessel  draft  or  more. 

I - — - — - — - 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

1 .1  Use  and  adjustment  of 
steering  controls. 

Ensures  that  heading  is 
maintained 

Operation  of  steering  controls  is 
consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

1 .2  Use  of  the  steering 
compass. 

Ensures  that  heading  is 
maintained 

Operation  of  steering  compass  is 
consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

1 .3  Use  of  the  rudder  angle 
indicator. 

Ensures  that  heading  is 
maintained 

Operation  of  rudder  angle  indicator  is 
consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

1 .4  Use  of  the  rate  of  turn 
indicator. 

Ensures  that  heading  is 
maintained 

Operation  of  rate  of  turn  indicator  is 
consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

1 .5  Use  of  steering  system 
alarms. 

Ensures  that  heading  is 
maintained 

Operation  of  steering  system  alarms 
is  consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

1 .6  Use  and  adjustment  of  the 
course  recorder. 

Ensures  that  heading  is 
maintained 

Operation  of  course  recorder  is 
consistent  with  manufacturer’s 
recommendations. 

□  Pass 

□  Fail 

2.1  Compare  and  report 
course  by  gyro  and 
magnetic  after  course 
change. 

Report  of  compass 
comparison 

Correct  report  after  course  change. 

□  Pass 

□  Fail 

2.2  Compare  and  report 
course  by  gyro  and 
magnetic  at  time  interval 
specified  by  stand  orders 
or  company  policy. 

Report  of  compass 
comparison 

Correct  report  at  the  time  interval 
specified  in  the  standing  orders  or 
company  policy  not  to  exceed  15 
minutes. 

□  Pass 

□  Fail 
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Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


Action 

Performance  Measure 

Performance  Standard 

Score 

2.3  Report  course  by  gyro  and 
magnetic  to  the  watch 
officer  when  changes  in 
compared  headings  are 
noted. 

Report  of  compass 
comparison 

Correct  report  when  changes  in 
compared  headings  are  noted. 

□  Pass 

□  Fail 

3.1  Repeat  helm  command  as 
given. 

Repetition  of  steering 
order 

Repeats  helm  command  word-for- 
word  using  standard  nomenclature 
and  terminology  in  compliance  with 
standing  orders  or  company  policy. 

□  Pass 

□  Fail 

3.2  Execute  course  change. 

Execution  of  course 
change 

Standard  is  to  perform  all  of  the 

following: 

□  Applies  appropriate  amount  of 
rudder  to  actuate  controllable 
turn  rate. 

□  Approaches  ordered  course, 
recognizes  turn  rate. 

□  Checks  swing,  applies  counter 
rudder  as  necessary. 

□  Steadies  up  on  ordered  course. 

□  Pass 

□  Fail 

3.3  Report  helm  order. 

Answers  question: 

What  is  the  chain  of 
command  among  the 
bridge  team? 

Report  as  soon  as  the  order  and/or 
action  has  been  completed. 

□  Pass 

□  Fail 

3.4  Understand  procedure  for 
conflicting  commands. 

Reports  chain  of 
command 

States  that  the  master’s  orders  are  to 
be  obeyed  whenever  there  is  conflict 
between  members  of  the  bridge 
team. 

□  Pass 

□  Fail 

4.1  Steer  by  gyrocompass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  ±  3°. 

□  Pass 

□  Fail 

4.2  Steer  by  magnetic 
compass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  ±  5°. 

□  Pass 

□  Fail 

4.3  Steer  by  visual  range  or 
landmark. 

Ensures  that  heading  is 
maintained 

Standard  is  to: 

□  Identify  range. 

□  Apply  rudder  to  correct  side. 

□  Maintain  forward  and  after  range 
in  a  vertical  line. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 

Section  3.  Additional  Assessor  Guidance 

•  The  assessor  should  give  the  candidate  at  least  five  helm  commands  during  this  part  of  the 
assessment. 

•  The  specified  steering  standards  may  be  modified  to  compensate  for  the  area  of  navigation 
and  prevailing  sea  state. 
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ASSESSMENT  WORKSHEET 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

5.  Demonstrate  the  use  of  magnetic  and  gyrocompasses  and  visual  range  in 

shallow  water  while  maneuvering. 

6.  Perform  steering  procedures  in  shallow  water  while  maneuvering. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  an  approved  steering  station  fitted  with  manual 
and  automatic  helm  modes,  gyro  repeater,  and  magnetic  compass.  Optimally,  the 
steering  system  should  have  manual  and  automatic  pilot,  non-follow-up  unit,  and 
navigation  modes  with  adjustable  controls,  rate  of  turn  indicator,  and  steering 
alarms. 

Initial  Condition 

This  assessment  should  occur  in  shallow  water  while  maneuvering.  Water  depth 
must  be  no  more  than  1 .5  times  vessel  draft. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

5.1  Repeat  helm  command 
as  given. 

Repeats  steering  order 

Repeats  helm  command  exactly 
using  standard  nomenclature  and 
terminology  in  compliance  with 
standing  orders  or  company  policy. 

□  Pass 

□  Fail 

5.2  Execute  course  change. 

Executes  course  change 

Standard  is  to: 

□  Apply  appropriate  amount  of 
rudder  to  actuate  controllable 
turn  rate. 

□  Approach  ordered  course, 
recognizes  turn  rate. 

□  Check  swing,  apply  counter 
rudder  as  necessary. 

□  Steady  up  on  ordered  course. 

□  Pass 

□  Fail 

5.3  Report  helm  order. 

Reports  steering  order 

Complies  with  standard  helm 
orders. 

□  Pass 

□  Fail 

5.4  Understand  procedure 
for  conflicting 
commands. 

Reports  of  chain  of 
command 

States  that  the  master’s  orders  are 
to  be  obeyed  whenever  there  is 
conflict  between  members  of  the 
bridge  team. 

□  Pass 

□  Fail 

6.1  Steer  by  gyrocompass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  ±  2°. 

□  Pass 

□  Fail 

6.2  Steer  by  magnetic 
compass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  ±  3°. 

□  Pass 

□  Fail 

6.3  Steer  by  visual  range  or 
landmark. 

Ensures  that  heading  is 
maintained 

Standard  is  to: 

□  Identify  range. 

□  Apply  rudder  to  correct  side. 

□  Maintain  forward  and  after 
range  in  a  vertical  line. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 

Appendix  B  of A  Method  for  Developing  Mariner  Assessments 


Assessment  Procedures/Helmsman  Duties 


ASSESSMENT  WORKSHEET  III 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

7.  Demonstrate  understanding  and  execution  of  helm  orders  while  steering  by 
visual  range  or  landmark. 

8  Perform  steering  procedures  while  steering  by  visual  range  or  landmark. 

Assessment  Method 

ShiDboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  an  approved  steering  station  fitted  with  manual 
and  automatic  helm  modes,  gyro  repeater,  and  magnetic  compass.  The  steering 
system  should  have  manual  and  automatic  pilot,  non-follow-up  unit,  and  navigation 
modes  with  adjustable  controls,  rate  of  turn  indicator,  and  steering  alarms. 

Initial  Condition 

This  assessment  should  occur  in  conditions  requiring  steering  by  visual  range  or 
landmark. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

7.1  Repeat  helm  command 
as  given. 

Repeats  steering  order 

Repeats  helm  command  exactly 
using  standard  nomenclature  and 
terminology  in  compliance  with 
standing  orders  or  company  policy. 

□  Pass 

□  Fail 

7.2  Execute  course  change. 

Executes  course  change 

□  Applies  appropriate  amount  of 
rudder  to  actuate  controllable 
turn  rate. 

□  Approaches  ordered  course, 
recognizes  turn  rate. 

□  Checks  swing,  applies  counter 
rudder  as  necessary. 

□  Steadies  up  on  ordered  course 

□  Pass 

□  Fail 

7.3  Report  helm  order. 

Reports  steering  order 

Complies  with  standard  helm 
orders. 

□  Pass 

□  Fail 

7.4  Understand  procedure 
for  conflicting 
commands. 

Reports  of  chain  of 
command 

States  that  the  master’s  orders  are 
to  be  obeyed  whenever  there  is 
conflict  between  members  of  the 
bridge  team. 

□  Pass 

□  Fail 

8.1  Steer  by  gyrocompass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  +  3°  (open 
waters). 

Maintains  heading  ±  2°  (shallow 
water). 

□  Pass 

□  Fail 

8.2  Steer  by  magnetic 
compass. 

Ensures  that  heading  is 
maintained 

Maintains  heading  ±  5°  (open 
waters). 

Maintains  heading  ±  3°  (shallow 
water). 

□  Pass 

□  Fail 

8.3  Steer  by  visual  range  or 
landmark. 

Ensures  that  heading  is 
maintained 

Standard  is  to : 

□  Identify  range. 

□  Apply  rudder  to  correct  side. 

□  Maintain  forward  and  after 
range  in  a  vertical  line. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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ASSESSMENT  WORKSHEET  IV 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

9.  Demonstrate  steering  duty  changeover. 

10.  Perform  helm  relief  procedures. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment,  oral  test  questions. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  an  approved  steering  station  fitted  with  manual 
and  automatic  helm  modes,  gyro  repeater  and  magnetic  compass.  Optimally,  the 
steering  system  should  have  a  manual,  automatic  pilot,  NFU,  and  navigation  modes 
with  adjustable  controls,  rate  of  turn  indicator,  and  steering  alarms. 

Initial  Condition 

This  assessment  can  be  conducted  in  any  of  the  following  conditions: 

•  Open  waters  at  sea  speed  (water  depth  =  6  times  draft  or  more). 

•  Shallow  water  while  maneuvering  (water  depth  =  1 .5  times  draft  or  less). 

•  Steering  by  visual  range  or  landmark. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

9.1  Changeover 

Ensures  changeover  of 

Changeover  procedure  is  consistent  with 

□ 

Pass 

steering  pumps. 

steering  pumps 

manufacturer's  recommendations. 

□ 

Fail 

9.2  Changeover 

Ensures  changeover  of 

Changeover  procedure  is  consistent  with 

□ 

Pass 

steering  motors. 

steering  motors 

manufacturer’s  recommendations. 

□ 

Fail 

9.3  Engage/ disengage 

Ensures  that  autopilot  is 

Changeover  procedure  is  consistent  with 

□ 

Pass 

autopilot. 

engaged/disengaged 

manufacturer’s  recommendations. 

□ 

Fail 

9.4  Engage/disengage 

Ensures  that  NFU  is 

Changeover  procedure  is  consistent  with 

□ 

Pass 

non-follow-up  unit. 

engaged/disengaged 

manufacturer’s  recommendations. 

□ 

Fail 

10.1  Hand  over  the 

Answers  question: 

Correct  report  includes  all  of  the  following: 

□ 

Pass 

helm. 

What  should  you  report 
when  handing  over  the 
helm  to  your  relief,  and 
to  whom  should  you 
report  this? 

Answers  question: 

What  position  should 
the  rudder  be  in  when 
the  helm  is  relieved? 

□  The  mode  of  operation  [hand,  autopilot, 
non-follow-up  unit  (NFU)]. 

□  Steering  unit  in  use  (port  or  starboard 
steering  unit). 

□  Vessel’s  heading: 

•  True  course. 

•  Gyro  course. 

•  Magnetic  compass  course. 

□  The  amount  of  helm  carried  to  maintain  a 
steady  course,  when  appropriate. 

Helmsman  who  has  been  relieved  should 
report  the  above  information  to  the  officer 
of  the  navigational  watch  after  being 
relieved. 

The  helm  should  be  relieved  with  no  rudder. 
Ensure  that  the  helmsman  checks  the  course 
recorder  chart  at  the  completion  of  his  or  her 
trick  on  the  wheel  whether  steering  by  hand, 
autopilot  or  non-follow-up  unit  (NFU). 

□ 

Fail 
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Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


10.2  Relieving  the 
wheel  during  a 
maneuver 

Answers  question: 

Under  what 
circumstances  should 
the  wheel  be  relieved 
during  a  maneuver? 

Correct  response  is  never .  Under  no 
circumstances  should  the  wheel  be  relieved 
during  a  maneuver. 

□  Pass 

□  Fail 

COMMENTS: 

FINAL  SCORE  □  Pass 

□  Fail 
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ASSESSMENT  WORKSHEET  V 


Section  1.  Assessment  Conditions 


Assessment  Objective 

1 1.  Demonstrate  helmsman  steering  failure  procedures. 

Assessment  Method 

Shipboard  assessment  and/or  ship  bridge  simulator  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  be  posted  at  an  approved  steering  station  fitted  with  manual 
and  automatic  helm  modes,  gyro  repeater,  magnetic  compass,  optimally  the 
steering  system  should  have  a  manual,  automatic  pilot,  NFU  and  navigation  modes 
with  adjustable  controls,  rate  of  turn  indicator,  and  steering  alarms. 

Initial  Condition 

This  assessment  should  be  conducted  in  open  waters  at  reduced  speed  (water 
depth  =  6  times  draft  or  more).  The  vessel  should  be  clear  of  navigational  hazards. 

A  steering  failure  drill  may  be  simulated  for  purpose  of  this  assessment.  Standing 
orders  or  company  policy  must  be  strictly  adhered  to  during  the  all  phases  of  the 
assessment. 

Section  2.  Action,  Performance  Measure,  Standards,  and  Score 


Action 

Performance  Measure 

Performance  Standard 

Score 

11.1  Respond  to  steering 
failure. 

Responds  to  steering 
failure 

□  Helmsman  reports  steering  failure 
to  the  officer  of  the  navigational 
watch. 

□  Alternative  and/or  emergency 
steering  system  is  engaged. 

□  Pass 

□  Fail 

COMMENTS:  FINAL  SCORE  □  Pass 

□  Fail 
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Appendix  C  to 

“A  Method  for  Developing  Mariner  Assessments” 


MARINER  ASSESSMENT  PROCEDURES  FOR  PREPARING 
THE  MAIN  ENGINE  FOR  OPERATION 


This  appendix  contains  example  procedures  for  assessing  a 
mariner’s  ability  to  prepare  the  main  engine  for  operation.  The 
assessment  package  consists  of  the  following:  assessor 
instructions,  candidate  instructions,  five  assessment  worksheets, 
and  an  assessment  control  sheet  for  documenting  the  final  results 
of  the  assessment. 


This  report  may  be  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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ASSESSOR  INSTRUCTIONS 


Introduction 

The  following  procedures  are  designed  for  assessing  a  candidate’s  ability  to  prepare  the  main 
engine  of  a  ship  for  operation.  This  includes  starting  the  main  engine  and  preparing  the  control 
system.  This  package  was  specifically  developed  for  a  propulsion  system  that  has  two  diesels 
driving  one  shaft  through  a  reduction  gear.  The  diesel  engines  described  in  the  assessment 
procedures  are  reversible,  medium  speed  engines.  With  some  modification,  this  package  may  be 
applicable  to  propulsion  plants  with  single,  slow  speed  diesel  engines. 

The  assessment  materials  include  Assessor  Instructions,  Candidate  Instructions,  an  Assessment 
Control  Sheet,  and  five  separate  assessment  modules  (Assessment  Worksheets  I-V).  The 
Assessor  Instructions  describe  the  assessment  objectives,  method,  performance  measures,  and 
performance  standards.  The  Assessor  Instructions  also  list  the  responsibilities  of  the  assessor 
during  each  phase  of  the  assessment  process.  The  Candidate  Instructions  explain  the  assessment 
process  from  the  candidate’s  perspective  and  define  the  responsibilities  of  the  candidate. 

The  Assessment  Control  Sheet  provides  a  record  of  the  names  of  the  candidate,  assessor,  and 
qualified  instructor  who  certified  that  the  candidate  met  the  training  prerequisites  for  assessment. 
The  control  sheet  also  includes  the  date  and  location  of  the  assessment  and  the  STCW  reference 
information  for  the  competency  area  being  assessed.  Section  3  of  the  Assessment  Control  Sheet 
provides  space  for  the  assessor  to  record  the  candidate’s  score  on  each  assessment  objective. 
Finally,  section  4  of  the  control  sheet  summarizes  the  assessment  objectives  and  the  actions 
comprising  each  objective. 

As  the  assessor,  you  will  evaluate  the  candidate’s  ability  to  complete  the  assessment  objectives 
correctly.  Using  the  scoring  procedures  described  below,  you  will  provide  the  candidate  with  a 
separate  score  for  each  assessment  objective  and  compile  these  scores  into  a  final  score  for  the 
entire  assessment.  The  scores  for  each  assessment  objective  and  the  candidate’s  overall  score  for 
the  entire  assessment  are  to  be  recorded  on  the  Assessment  Control  Sheet. 

The  rest  of  this  section  briefly  describes  the  various  components  of  this  assessment.  Assessment 
Worksheets  I  through  V  provide  additional  information  about  the  assessment  objectives,  method, 
conditions,  performance  measures,  performance  standards,  and  scoring  procedures. 

Assessment  Objectives 

In  a  step-by-step  procedure,  the  candidate  should  safely  start  the  main  engine.  This  includes  the 
following  steps: 

1 .  Perform  pre-start  checks. 

2.  Roll  over  the  main  engine. 

3.  Start  the  engine  ahead  and  astern. 

4.  Transfer  control  to  a  remote  operating  station. 

5.  Prepare  the  propulsion  plant  to  answer  bells. 
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Assessment  Method 

Although  starting  procedures  can  be  practiced  in  a  simulator  and  knowledge  of  procedures  for 
starting  the  main  engine  can  be  evaluated  by  written  examination,  the  final  assessment  for  this 
proficiency  should  be  conducted  onboard  a  vessel.  This  ensures  the  candidate  can  perform  the 
tasks  in  an  actual  work  setting. 

Assessment  Conditions 

This  assessment  should  begin  at  the  local  engine  control  station.  The  candidate  should  then  work 
throughout  the  engine  room.  As  the  assessor,  you  should  tell  the  candidate  each  subsequent 
activity  to  complete,  but  not  the  individual  actions  necessary  to  complete  the  activity.  The 
candidate  will  not  receive  any  assistance  from  others,  except  to  check  pressure  and  temperature 
levels  as  required  during  the  assessment.  The  conditions  you  can  vary  during  the  assessment  are 
specified  in  the  next  section  under  “Preparing  for  the  Assessment”  and  on  the  individual 
worksheets  for  each  assessment  objective. 

Performance  Measures  and  Standards 

Each  assessment  objective  is  comprised  of  one  or  more  actions.  Each  action  has  one  or  more 
corresponding  performance  measures  and  standards.  For  example,  assessment  objective  2 
requires  six  different  actions  (see  section  2  of  Assessment  Worksheet  II  on  p.  15).  Note  that 
within  one  action,  such  as  action  2.6,  there  may  be  more  than  one  performance  measure.  In  the 
case  of  action  2.6,  the  candidate  has  to  pass  both  measures  in  order  to  pass  the  action.  If  the 
candidate  fails  one  measure,  he  or  she  fails  the  action. 

Use  pass/fail  scoring  for  each  assessment  objective: 

□  Pass  -  The  candidate’s  total  score  was  at  or  above  the  passing  level  noted  at  the  bottom  of 
each  Assessment  Worksheet.  The  candidate  also  correctly  performed  all  “required”  actions  - 
that  is,  those  that  would  result  in  an  automatic  failure  if  performed  incorrectly.  Required 
actions  are  noted  in  the  “standards”  column  of  each  Assessment  Worksheet. 

□  Fail  -  The  candidate  either  did  not  have  a  high  enough  score  or  did  not  properly  complete  all 
of  the  required  items. 

Note  that,  if  a  candidate  fails  an  assessment  objective,  he  or  she  automatically  fails  the  entire 
assessment.  If  this  occurs,  you  should  terminate  the  assessment  without  proceeding  to  the  next 
objective. 

Individual  measures  and  standards  for  each  assessment  objective  are  specified  on  Assessment 
Worksheets  I  through  V. 

Assessment  Checklist 

The  following  checklist  summarizes  the  tasks  and  responsibilities  of  the  assessor  at  each  phase  of 
the  assessment  process. 
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Preparing  for  the  Assessment 

□  Inform  the  engineering  watch  officer,  bridge  watch  officer,  and  candidate(s)  of  the  date  and 
time  of  the  assessment. 

□  Obtain  communication  equipment  for  yourself  and  the  candidate(s). 

□  Check  the  condition  of  each  of  the  possible  variables  for  each  assessment  objective.  Then, 
set  each  variable  to  any  level  you  desire,  within  a  normal  range.  The  intent  is  to  set  up  the 
conditions  so  that  a  candidate  must  demonstrate  knowledge  of  the  main  engine  systems  -  not 
just  memorize  a  series  of  steps.  This  also  helps  to  vary  the  assessment  from  candidate  to 
candidate.  Here  are  the  conditions  you  can  vary  before  each  assessment: 

Assessment  Objective  1 

•  Jacket  water  head  tank  level. 

•  Clutch  air  control-valve  position. 

•  Cylinder  air  cock  positions. 

•  Starting  air  supply  to  the  engine. 

•  Emergency  stop  switch  position. 

Assessment  Objective  2 

•  Power  to  the  jacket  water  pump. 

•  Power  to  the  lubricating  oil  pump. 

•  Presence  of  leak(s)  in  the  jacket  water  system. 

•  Relief  valve  setting  on  the  lubricating  oil  pump. 

Assessment  Objective  3 

•  Fuel  oil  pressure  drop  across  the  suction  strainer. 

•  Fuel  oil  supply/retum  solenoid  valve  positions. 

Assessment  Objective  4 

•  Location  of  engine  control  (it  can  be  at  either  a  remote  engineering  station  or  the 
bridge  station,  but  the  engine  should  be  in  a  local  mode  of  operation  before 
beginning  the  assessment  of  this  objective). 

Assessment  Objective  5 

•  Settings  on  the  automatic  control  (this  is  to  test  the  candidates’  knowledge  of  the 
control  system). 

□  For  all  assessment  objectives,  ensure  that  the  reduction  gear,  thrust  bearings,  and  stem  seal 
systems  are  set  up  properly. 

□  Determine  the  appropriate  operating  conditions  for  the  ship’s: 

•  Lube  oil  temperature  (e.g.,  normal  starting  range  is  130-150°  F). 

•  Lube  oil  pressure. 
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•  Starting  air  pressure. 

•  Jacket  water  temperature. 

•  Jacket  water  pump  pressure. 

•  Fuel  pump  pressure. 

□  Remember  that  this  assessment  should  take  place  under  normal  conditions.  If  abnormal 
conditions  arise,  suspend  the  assessment  and  restart  after  you  have  corrected  any  abnormal 
conditions. 

Briefing  the  Candidate  before  the  Assessment 

□  This  assessment  should  be  conducted  for  two  candidates  at  once  -  each  candidate  should  be 
responsible  for  starting  one  of  the  main  engines. 

□  Provide  each  candidate  with  a  copy  of  the  Candidate  Instructions  and  Assessment  Control 
Sheet. 

□  Review  the  staring  main  engine  assessment  instructions  with  the  candidates  and  answer  any 
questions. 

□  Explain  the  assessment  performance  measures  and  standards. 

□  Discuss  the  desired  outcome(s)  and  consequences  of  failing  to  perform  part  of  all  of  the 
assessment. 

□  Explain  that  the  assessment  will  be  terminated  if: 

•  The  candidate  does  not  correctly  perform  all  of  the  required  actions. 

•  One  or  more  of  the  initial  equipment  conditions  become  abnormal  or  unsafe. 

•  The  candidate  violates  vessel  safety  procedures. 

□  Remind  the  candidates  that  it  is  permissible  to  ask  questions  during  the  assessment, 
especially  if  they  have  a  safety  concern. 

□  Advise  the  candidates  that  they  may  have  assistants  to  help  with  activities  that  take  time  such 
as  opening  the  blow  down  valves.  However,  the  candidates  are  responsible  for  directing  the 
assistants’  activities  and  verifying  that  the  assistants’  tasks  have  been  carried  out  correctly. 

□  Ensure  that  the  candidates  have  the  proper  equipment  to  carry  out  the  assessment: 

•  Small  notebook  and  pencil  for  writing  down  conditions  and  readings. 

•  Wrenches  used  for  opening  valves. 

•  Long-sleeved  shirt  to  prevent  burning  arms  when  reaching  over  hot  pipes. 

•  Pair  of  work  gloves. 

□  Discuss  the  candidates’  willingness  to  be  assessed  under  the  circumstances  presented. 
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Observing  the  Candidate’s  Performance 

□  If  a  safety  violation  occurs,  terminate  the  assessment  immediately. 

□  Continuously  observe  the  candidates  during  the  assessment.  Require  that  standard 
procedures  or  company  policy  be  adhered  to  except  when  assessment  procedures  require 
demonstration  of  knowledge  or  skill  different  from  a  convention  adopted  by  the  company  or 
facility. 

□  Ensure  realistic  assessment  conditions  consistent  with  a  normal  working  environment  for  a 
marine  engineer.  Ensure  the  candidates  can  concentrate  on  the  task  at  hand. 

□  Avoid  giving  the  candidates  unsolicited  assistance,  but  respond  to  appropriate  questions  and 
provide  appropriate  equipment  when  required. 

□  Remain  objective  and  maintain  positive  control  of  the  operation  at  all  times. 

Determining  Assessment  Outcome 

□  Record  performance  on  the  appropriate  assessment  worksheets. 

□  Strictly  adhere  to  the  prescribed  scoring  procedures. 

□  Remember  that  if  either  candidate  incorrectly  performs  any  of  the  required  actions,  he  or  she 
automatically  fails  the  entire  assessment. 

□  Determine  and  document  the  outcome  of  the  assessment,  then  transfer  the  final  results  to  the 
Assessment  Control  Sheet. 

Debriefing  the  Candidate 

□  Debrief  each  candidate  as  soon  as  possible  after  the  assessment. 

□  Provide  each  candidate  with  a  copy  of  the  Assessment  Control  Sheet. 

□  Restate  the  assessment  objectives. 

□  Focus  on  positive  accomplishments  first. 

□  Identify  areas  needing  improvement. 

□  If  the  candidate  failed  to  demonstrate  proficiency,  jointly  develop  an  improvement  plan  to 
prepare  for  reassessment. 
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CANDIDATE  INSTRUCTIONS 


In  this  assessment,  you  will  be  evaluated  on  your  ability  to  safely  prepare  the  main  engine  for 
operation.  Table  1  shows  the  1 995  STCW  Code  specification  for  preparation  of  main  and 
auxiliary  machinery. 


Table  1.  STCW  Code  specification  for  preparation  of  main  and  auxiliary  machinery. 


STCW  Requirement 

1995  STCW  Code,  Section  A-lll/1  (p.  75):  Mandatory  minimum 
requirements  for  certification  of  ratings  for  officers  in  charge  of  an 
engineering  watch  in  a  manned  engine  room  or  designated  duty  engineers 
in  a  periodically  unmanned  engine  room. 

STCW  Function 

Marine  engineering  at  the  operational  level. 

STCW  Competence 

Operate  main  and  auxiliary  machinery  and  associated  control  systems. 

STCW  Proficiency 

Preparation  of  main  machinery  and  preparation  of  auxiliary  machinery  for 
operation. 

Assessment  Objectives  and  Methods 

A  qualified  assessor  will  assess  your  ability  to  meet  the  five  assessment  objectives  in  the  first 
column  of  Table  2.  Note  that  each  assessment  objective  has  a  corresponding  assessment  type. 
Performance  assessed  by  “practical  skill  demonstration”  means  your  assessor  will  ask  you  to 
demonstrate  your  ability  to  perform  the  actions  required  in  the  objective.  Your  skill 
demonstration  will  occur  onboard  ship.  Performance  assessed  by  “oral  or  written  test  questions” 
means  your  assessor  will  ask  you  to  respond  to  one  or  more  questions  and  then  assess  your 
ability  to  respond  correctly  to  each  question. 


Table  2.  Assessment  objectives  and  methods  for  main  engine  preparation. 


Assessment  Objective 

Assessment  Method 

1 .  Perform  pre-start  checks. 

Practical  skill  demonstration  aboard  ship  /  Oral  or  written 
test  questions 

2.  Roll  over  the  main  engine. 

Practical  skill  demonstration  aboard  ship  /  Oral  or  written 
test  questions 

3.  Start  the  main  engine  locally. 

Practical  skill  demonstration  aboard  ship  /  Oral  or  written 
test  questions 

4.  Transfer  main  engine  controls. 

Practical  skill  demonstration  aboard  ship 

5.  Prepare  plant  to  answer  bells. 

Practical  skill  demonstration  aboard  ship  /  Oral  or  written 
test  questions 

Refer  to  section  4  of  the  Assessment  Control  Sheet  for  a  list  of  the  specific  actions  required  for 
each  objective  (you  should  receive  a  copy  of  this  document  from  the  assessor  during  your  pre- 
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assessment  briefing).  Below  are  some  general  guidelines  for  what  you  should  expect  during  the 
assessment  process. 

Pre- Assessment  Briefing  with  Assessor 

This  briefing  should  occur  approximately  a  week  in  advance  of  the  scheduled  assessment,  if 
possible.  This  will  help  you  and  the  assessor  to  be  well  prepared  for  the  assessment  when  it 
actually  occurs.  During  this  briefing,  you  should: 

•  Discuss  your  prior  experience,  training,  and/or  company  policy  with  the  assessor.  On  the 
basis  of  these  qualifications,  discuss  whether  you  are  qualified  to  undertake  this 
assessment.  If  you  both  agree  you  are  qualified,  then  continue  with  the  assessment 
process.  If  not,  arrange  for  additional  on-the-job  or  simulator  training,  and  set  a  date  for 
another  review  of  your  qualifications. 

•  Obtain  a  copy  of  the  Assessment  Control  Sheet  from  your  assessor.  Review  this 
document  and  discuss  the  scope  and  depth  of  knowledge  covered  by  this  assessment. 

Ask  questions  about  any  part  of  the  assessment  that  is  unclear. 

•  Review  the  assessment  performance  measures  and  standards,  and  ask  any  questions  you 
have  about  them. 

•  Discuss  the  desired  outcome(s)  and  the  consequences  of  failing  to  perform  any  part  of  the 
assessment. 

•  Discuss  the  general  assessment  procedures.  Your  assessor  will  inform  you  of  how  much 
time  is  allowed,  when  the  assessment  begins,  and  under  what  circumstances  he  or  she 
will  terminate  the  assessment. 

•  Discuss  any  non-standard  procedures  that  are  expected  of  you  during  the  assessment. 

•  Consider  whether  you  are  willing  to  be  assessed  under  the  circumstances  presented  and 
advise  the  assessor  of  your  willingness  to  undertake  the  assessment. 

Participation  in  the  Assessment 

Your  assessment  will  begin  at  the  local  engineering  control  station.  During  the  assessment,  you 
will  be  working  throughout  the  engine  room,  including  a  remote  engine  control  station  and/or  the 
bridge  control  station. 

The  assessor  will  suspend  the  assessment  if  abnormal  conditions  develop  with  any  equipment  or 
if  the  safety  of  personnel  or  equipment  is  endangered.  He  or  she  will  also  suspend  the 
assessment  if  you  fail  to  correctly  perform  one  of  the  “required”  actions  for  this  assessment 
(these  actions  are  italicized  on  the  Assessment  Worksheets  in  the  “standard”  column). 

You  may  have  assistants  to  help  with  activities  that  take  time  such  as  opening  the  blow  down 
valves.  You  are  responsible  for  directing  your  assistants’  activities  and  verifying  that  their  tasks 
have  been  carried  out  correctly. 

Your  assessor  will  continuously  observe  you  during  the  assessment.  You  are  expected  to  adhere 
to  standard  procedures  or  company  policy  unless  the  assessor  briefs  you  on  a  requirement  to 
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perform  a  non-standard  procedure.  During  the  assessment,  remember  to: 

•  Use  appropriate  marine  terminology  and  nomenclature  at  all  times. 

•  Listen  to  an  entire  question  before  responding  or  acting.  Remember  that  you  may  not  use 
reference  material  of  any  kind  during  the  assessment  examination. 

•  Ask  questions  if  you  have  a  safety  concern.  The  assessor  will  provide  you  with 
appropriate  responses  to  your  questions.  If  a  safety  violation  occurs,  the  assessor  will 
terminate  the  assessment  immediately. 

Your  assessor  will  ensure  realistic  assessment  conditions  consistent  with  the  normal  working 
environment  for  an  engineer. 

The  Outcome  of  Your  Assessment 

The  assessor  will  record  your  performance  on  a  series  of  Assessment  Worksheets.  He  or  she  will 
score  each  assessment  objective  on  a  “Pass/Fail”  basis.  Acceptable  performance  (a  “Pass”  score) 
will  be  based  on  your  ability  to  correctly  respond  to  examination  questions.  It  will  also  be  based 
on  your  ability  to  perform  assigned  tasks  safely  in  a  manner  that  demonstrates: 

•  The  required  level  of  skill,  knowledge,  and  ability. 

•  Sound  and  professional  judgment. 

Unacceptable  performance  (a  “Fail”  score)  will  be  based  on  your  failure  to  perform  a  critical 
phase  of  the  assessment  exercise  proficiently.  When  this  occurs,  your  assessment  will  be 
suspended  and  postponed  until  you  receive  further  instruction  and  training. 

Note  that  the  following  events  will  terminate  an  assessment  immediately: 

•  An  action,  or  lack  of  action,  by  you  which  required  corrective  action  or  intervention  by 
the  assessor  to  prevent  injury,  damage,  or  the  development  of  a  hazardous  condition. 

•  Failing  to  use  proper  procedures,  including  appropriate  communication  procedures, 
during  the  assessment. 

•  Failing  to  take  prompt  corrective  action  when  required. 

The  assessor  will  strictly  adhere  to  pre-determined  scoring  procedures,  and  will  document  the 
outcome  of  the  assessment.  He  or  she  will  transfer  the  final  results  of  your  assessment  to  the 
Assessment  Control  Sheet. 
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Assessment  Debriefing 

You  and  the  assessor  should  discuss  the  assessment  results  as  soon  as  possible  after  the 
assessment.  During  the  debriefing: 

•  The  assessor  should  provide  you  with  a  copy  of  the  Assessment  Control  Sheet  describing 
the  results  of  your  main  engine  preparation  assessment. 

•  The  assessor  should  restate  the  assessment  objective(s)  and  identify  those  that  you 
successfully  demonstrated. 

•  If  appropriate,  you  and  the  assessor  should  discuss  the  areas  in  which  you  need 
improvement,  and  then  develop  an  improvement  plan  based  on  the  assessment  outcome. 
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ASSESSMENT  CONTROL  SHEET 


Section  1.  Assessment  Reference  Information 


1.  Name  of  Candidate 

2.  Name  of  Designated  Assessor 

3.  Signature  of  Qualified  Instructor1 

4.  Date  of  Assessment 

5.  Assessment  Location2 

Section  2.  STCW  Reference  Information 


Assessment  Area 

Preparing  the  main  diesel  engine  for  operation. 

Assessment  Method 

Shipboard  assessment. 

STCW  Requirement 

STCW  Code,  Section  A-lll/1 ,  p.  75  -  Mandatory  minimum  requirements  for 
certification  of  ratings  for  officers  in  charge  of  an  engineering  watch  in  a  manned 
engine  room  or  designated  duty  engineers  in  a  periodically  unmanned  engine 
room. 

STCW  Function 

Marine  engineering  at  the  operational  level. 

STCW  Competence 

Operate  main  and  auxiliary  machinery  and  associated  control  systems. 

STCW  Proficiency 

Preparation  of  main  machinery  for  operation. 

Section  3.  Assessment  Objectives,  Methods,  and  Scores 


Assessment  Objective 

Assessment  Method 

Score 

1 .  Perform  pre-start  checks. 

Practical  skill  demonstration  aboard  ship  /  Oral 
test  questions 

□  Pass  □  Fail 

2.  Roll  over  the  main  engine. 

Practical  skill  demonstration  aboard  ship  /  Oral 
test  questions 

□  Pass  □  Fail 

3.  Start  the  main  engine  locally. 

Practical  skill  demonstration  aboard  ship  /  Oral 
test  questions 

□  Pass  □  Fail 

4.  Transfer  main  engine  controls. 

Practical  skill  demonstration  aboard  ship 

□  Pass  □  Fail 

5.  Prepare  plant  to  answer  bells. 

Practical  skill  demonstration  aboard  ship  /  Oral 
test  questions 

O  Pass  □  Fail 

COMMENTS  FINAL  SCORE  □  Pass  □  Fail 

1  The  qualified  instructor  certifies  that  the  candidate  has  met  the  training  prerequisites. 

2  Write  the  name  of  the  simulator  or  vessel  and  her  location  during  the  assessment  (e.g., 
Massachusetts  Maritime  Academy  training  simulator,  S/R  Long  Beach  -  Day  1  of  Puget  Sound 
to  Valdez  run,  etc.). 
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Section  4.  Assessment  Worksheet  Summary 


Assessment 

Worksheet 

Assessment  Objective 

Action 

1 

1 .  Perform  pre-start  checks. 

1.1 

Check  power  to  jacket  water  pump. 

1.2 

Check  power  to  lube  oil  pump. 

1.3 

Check  power  to  fuel  booster  pump. 

1.4 

Check  shaft  turning  gear. 

1.5 

Check  engine  barring  gear. 

1.6 

Check  for  local  control  of  engine. 

1.7 

Ensure  air  cocks  are  open. 

1.8 

Knowledge  of  proper  lube  oil  temperature. 

1.9 

Check  lubricating  oil  sump  temperature. 

1.10 

Knowledge  of  proper  air  pressure. 

1.11 

Check  starting  air  pressure. 

1.12 

Check  fuel  oil  day  tank  level. 

1.13 

Knowledge  of  proper  fuel  oil  level. 

1.14 

Check  level  of  jacket  water  head  tank. 

1.15 

Knowledge  of  how  to  fill  jacket  water  head  tank. 

1.16 

Check  emergency  stop  switch  position. 

1.17 

Check  position  of  the  air  supply  to  clutch. 

II 

2.  Roll  over  the  main  engine. 

2.1 

Start  /  check  jacket  water  pump  pressure. 

2.2 

Knowledge  of  proper  jacket  water  pump 

pressure. 

2.3 

Start  /  check  lubricating  oil  pressure. 

2.4 

Knowledge  of  proper  lubricating  oil  pressure. 

2.5 

Initiate  roll  over. 

2.6 

Check  condition  of  cylinders  during  engine 

rollover. 

III 

3.  Start  the  main  engines 

3.1 

Check  fuel  oil  supply  /  return  solenoids. 

locally  in  an  idling  condition 

3.2 

Start  /  check  fuel  oil  booster  pump  pressure. 

in  both  the  ahead  and 

3.3 

Knowledge  of  proper  fuel  oil  booster  pump 

astern  directions. 

pressure. 

3.4 

Check  fuel  oil  booster  pump  strainer  differential 

pressure. 

3.5 

Knowledge  of  proper  fuel  oil  booster  pump 

strainer  differential  pressure. 

3.6 

Check  emergency  stop  valve.  Place  in  the  run 

position. 

3.7 

Start  engine  in  ahead  direction. 

3.8 

Check  lubricating  pressure  upon  starting 

engine. 

3.9 

Inspect  engine  while  running. 

3.10  Stop  engine. 

3.11 

Start  engine  in  astern  direction. 

3.12  Stop  engine. 

IV 

4.  Transfer  control  of  the  main 

4.1 

Switch  control  of  engine  from  local  console  to 

engine  to  a  remote 

engineer’s  operating  station. 

operating  station. 

4.2 

Start  engine  in  ahead  direction. 

4.3 

Start  engine  in  astern  direction. 

V 

5.  Prepare  plant  to  answer 

5.1 

Open  clutch  control  valve. 

bells. 

5.2 

Ensure  proper  set  up  of  automatic  controls. 

5.3  Advise  bridge  that  the  propulsion  plant  is  ready. 
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ASSESSMENT  WORKSHEET  I 


Section  1.  Assessment  Conditions 


Assessment  Objective 

1.  Perform  pre-start  checks. 

Assessment  Method 

Shipboard  assessment  and  oral  test  questions. 

Candidate  Orientation 

There  should  be  two  candidates  for  assessment,  each  in  charge  of  starting  one 
engine.  The  assessor  briefs  the  candidates  on  assessment  methods,  conditions,  and 
standards.  Each  candidate  may  have  assistants  to  help  with  some  tasks,  such  as 
opening  and  closing  blow  down  valves.  Each  candidate  must  demonstrate  that  he  or 
she  is  in  charge  of  the  assistants  and  knows  the  tasks  and  when  they  are  completed. 
The  assessor  should  tell  the  candidates  when  to  begin  the  tasks  and  he  or  she 
should  check  that  each  task  is  properly  completed. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidates  should  begin  the  assessment  at  the  engineer’s  local  operating 
console.  Main  engine  and  supporting  equipment  should  be  secured  but  operational 
and  available  for  use.  Reduction  gear  and  shaft  system  should  be  lined  up  and 
operational. 

Initial  Conditions 

The  assessor  should  have  received  clearance  from  both  the  bridge  and  engineering 
watch  officer  to  begin  the  assessment.  The  area  should  be  clear  so  the  assessor  can 
observe  activities.  This  assessment  is  an  evaluation  of  the  candidate’s  ability  to  start 
the  main  engine  under  normal  conditions.  Abnormal  conditions  will  not  be 
introduced.  If  abnormal  conditions  arise,  the  assessment  will  be  suspended  until  the 
condition  is  corrected.  The  assessor  should  vary  the  following  conditions  within  a 
normal  range: 

•  Jacket  water  head  tank  level. 

•  Clutch  air  control-valve  position. 

•  Cylinder  air  cock  positions. 

•  Starting  air  supply  to  the  engine. 

•  Emergency  stop  switch  position. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

1.1  Check  power  to 
jacket  water 
pump. 

Report  of  availability  of  power 

Correct  report  of  power  to  jacket 
water  pump 

□  Pass 

□  Fail 

1.2  Check  power  to 
lube  oil  pump. 

Report  of  availability  of  power 

Correct  report  of  power  to  lube  oil 
pump 

□  Pass 

□  Fail 

1.3  Check  power  to 
fuel  booster  pump. 

Report  of  availability  of  power 

Correct  report  of  power  to  fuel 
booster  pump 

□  Pass 

□  Fail 

1.4  Check  shaft 
turning  gear. 

Ensure  turning  gear  is 
disengaged 

Turning  gear  correctly  disengaged 
and  status  correctly  reported 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

1.5  Check  engine 
barring  gear. 

Ensure  barring  gear  is 
disengaged 

Barring  gear  correctly  disengaged 
and  status  correctly  reported 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 
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Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


1.6  Check  for  local 
control  of  engine. 

Ensure  engines  are  in  local 
control 

Engines  placed  in  local  control  and 
status  correctly  reported 

□  Pass 

□  Fail 

1.7  Ensure  air  cocks 
are  open. 

Report  that  air  cocks  are  open 

Correct  report  of  air  cock  position 

□  Pass 

□  Fail 

1 .8  Knowledge  of 
proper  lube  oil 
temperature. 

Response  to  question: 

What  should  the  engine  oil 
temperature  be  before 
initiating  engine  start  up? 

Correct  response  is  that  a  normal 
starting  range  for  the  engine  oil 
temperature  is  120°  F  to  150°  F 

□  Pass 

□  Fail 

1.9  Check  lubricating 
oil  sump 
temperature. 

Response  to  question: 

What  is  the  current  lubricating 
oil  sump  temperature? 

Correct  report  of  actual 
temperature 

□  Pass 

□  Fail 

1.10  Knowledge  of 
proper  starting 
air  pressure. 

Response  to  question: 

What  should  the  starting  air 
pressure  be  before  initiating 
engine  start  up? 

Correct  response  is  that  starting  air 
pressure  should  be  at  least  250  psi 

□  Pass 

□  Fail 

1.11  Check  starting  air 
pressure. 

Response  to  question: 

What  is  the  current  pressure? 

Correct  report  of  actual  pressure 

□  Pass 

□  Fail 

1.12  Check  fuel  oil 
day  tank  level. 

Response  to  question: 

What  is  the  current  level  of  fuel 
in  the  day  tank? 

Correct  report  of  fuel  in  the  day 
tank 

□  Pass 

□  Fail 

1.13  Knowledge  of 
proper  fuel  oil 
level. 

Response  to  question: 

What  is  the  minimum  level  of 
fuel  allowable  in  the  day  tank? 

Correct  response  is  that  the  day 
tank  should  be  at  least  Vz  full 

□  Pass 

□  Fail 

1.14  Check  level  of 
jacket  water 
head  tank. 

Response  to  question: 

What  is  the  current  level  of 
jacket  water  in  the  head  tank? 

Correct  report  of  level  in  the  jacket 
water  head  tank 

□  Pass 

□  Fail 

1.15  Knowledge  of 
howto  fill  jacket 
water  head  tank. 

Response  to  question: 

How  do  you  fill  it  if  it  is  too 
low? 

Knowledge  of  how  to  fill  the  tank 

□  Pass 

□  Fail 

1.16  Check 

emergency  stop 
switch  position. 

Response  to  question: 

In  what  position  is  the 
emergency  stop  switch? 

Correct  report  of  position  of 
emergency  stop  switch. 

Incorrect  report  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

1.17  Check  position  of 
the  air  supply  to 
clutch. 

Response  to  question: 

What  is  the  status  of  the  air 
supply  to  clutch? 

Correct  report  on  status  of  the  air  to 
the  clutch 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  □  Pass 

Total  the  number  of  “Pass”  scores  you  indicated  above.  □  Fail 

Pass:  Score  of  12-17  Fail:  Score  of  0-11 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet _ 
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ASSESSMENT  WORKSHEET 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

2.  Roll  over  the  main  engine. 

Assessment  Method 

Shipboard  assessment  and  oral  test  questions. 

Candidate  Orientation 

There  should  be  two  candidates  for  assessment,  each  in  charge  of  starting  one 
engine.  Assessor  briefs  the  candidates  on  assessment  methods,  conditions,  and 
standards.  Each  candidate  must  demonstrate  his  or  her  ability  to  “roll-over”  the 
main  engine  for  a  pre-start  inspection  for  leaky  cylinder  liners. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  the  assessment  at  the  engineer’s  local  operating 
console.  The  main  engine  and  supporting  equipment  should  be  secured  but 
operational  and  available  for  use. 

Initial  Condition 

Please  note  the  comments  regarding  abnormal  conditions  on  Assessment 
Worksheet  1.  Before  this  assessment  begins,  the  assessor  should  be  able  to  vary: 

•  Power  to  the  jacket  water  pump. 

•  Power  to  the  lubricating  oil  pump. 

•  Presence  of  leak(s)  in  the  jacket  water  system. 

•  Relief  valve  setting  on  the  lubricating  oil  pump. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

2.1  Start  /  check  jacket 
water  temperature. 

Answer  question: 

What  is  the  current  jacket 
water  temperature? 

Correct  report  of  current  jacket 
water  temperature 

a  Pass 

□  Fail 

2.2  Knowledge  of  proper 
jacket  water 
temperature. 

Answer  question: 

What  is  the  proper  jacket  water 
temperature? 

Correct  response  is  that  jacket 
water  temperature  should  be  a 
minimum  of  130°  F.  An  acceptable 
range  for  starting  is  140-160°  F. 

□  Pass 

□  Fail 

2.3  Start /check 
lubricating  oil 
pressure. 

Answer  question: 

What  is  the  current  lubricating 
oil  pressure? 

Correct  report  of  current  lubricating 
oil  pressure 

□  Pass 

□  Fail 

2.4  Knowledge  of  proper 
lubricating  oil 
pressure. 

Answer  question: 

What  is  the  proper  lube  oil 
pressure? 

Correct  response  is  the  lube  oil 
pressure  should  be  a  minimum  of 

25  psi 

□  Pass 

□  Fail 

2.5  Initiate  rollover. 

Report  on  successful  rollover 

Correct  procedure  used 

□  Pass 

□  Fail 
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Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


2.6  Check  condition  of 
cylinders  during 
engine  rollover. 


Report  on  condition  of 
cylinders  as  engine  is  “rolled 
over.” 

Answer  questions: 

What  is  the  significance  of 
water  coming  out  of  the  blow 
down  valves? 


What  is  the  significance  of  lube 
oil  coming  out  of  the  blow 
down  valves? 


Correct  report  of  condition  of 

cylinders  during  engine  rollover 

Correct  responses  to  question: 

□  A  water  leak  means  there  is  a 
jacket  water  leak  into  the 
cylinder.  This  is  a  problem 
because  water  is  not 
compressible,  so  when  the 
engine  is  started,  the  water  will 
damage  the  valves  and  piston. 
Also,  a  major  water  leak  will 
result  in  improper  engine 
cooling. 

□  The  significance  of  lube  oil 
coming  out  of  the  blow  down 
valves  depends  on  the  amount 
of  oil.  A  small  amount  is 
normal.  If  oil  continues  to 
come  out  as  you  roll  the 
engine  over,  there  is  an  oil 
seal  failure  in  the  turbo  charger 
or  air  blower.  This  is 
dangerous  as  it  can  cause  the 
engine  to  overspeed. 


- 1 -  1 

SCORING  PROCEDURE 

FINAL  SCORE 

□  Pass 

Total  the  number  of  “Pass"  scores  you  indicated  above. 

□  Fail 

Pass:  Score  of  5-6  Fail:  Score  of  0-4 

Transfer  the  final  score  at  right  to  the  Assessment  Contrql  Sheet. 

□  Pass 

□  Fail 
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ASSESSMENT  WORKSHEET 


Section  1.  Assessment  Conditions 


Assessment  Objective 

3.  Start  the  main  engines  locally  in  an  idling  condition  in  both  the  ahead  and 
astern  directions. 

Assessment  Method 

Shipboard  assessment  and  oral  test  questions. 

Candidate  Orientation 

There  should  be  two  candidates,  one  in  charge  of  starting  each  engine.  The 
assessor  briefs  the  candidates  on  assessment  methods,  conditions,  and 
standards. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  the  assessment  at  the  engineer’s  local  control  station. 

Initial  Condition 

The  main  engine  should  have  been  successfully  “rolled  over.”  All  operating 
requirements  at  this  point  are  satisfactory,  including  levels,  temperatures,  and 
pressures  associated  with  support  systems  including  jacket  water,  lubricating  oil, 
and  air  systems.  If  abnormal  conditions  arise,  the  assessment  should  be 
suspended.  The  assessor  should  be  able  to  vary: 

•  Fuel  oil  pressures  drop  across  the  suction  strainer. 

•  Fuel  oil  supply/return  solenoid  valve  positions. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

3.1  Check  fuel  oil  supply  / 
return  solenoids. 

Answer  question: 

What  is  the  status  of  solenoid 
valves? 

Correct  report  of  status  of  solenoid 
valves. 

Incorrect  report  of  fuel  oil  solenoid 
positions  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

3.2  Start  /  check  fuel  oil 
booster  pump 
pressure. 

Answer  question: 

What  is  the  current  fuel  oil 
pressure? 

Correct  report  of  current  pressure 

□  Pass 

□  Fail 

3.3  Knowledge  of  proper 
fuel  oil  pump 
pressure. 

Answer  question: 

What  is  the  minimum  fuel  oil 
pump  pressure? 

Correct  response  is  that  the 
minimum  fuel  oil  pump  pressure  is 

30  psi 

□  Pass 

□  Fail 

3.4  Check  fuel  oil  booster 
pump  strainer 
differential  pressure. 

Answer  question: 

What  is  the  current  fuel  oil 
pump  strainer  differential 
pressure? 

Correct  report  of  current  differential 
pressure 

□  Pass 

□  Fail 

3.5  Knowledge  of  proper 
fuel  oil  booster  pump 
strainer  differential 
pressure. 

Answer  question: 

What  is  the  significance  of  too 
high  a  differential  pressure? 

Correct  response  is  that  too  high  a 
differential  pressure  would  mean 
that  the  system  is  not  set  up 
properly;  a  valve  is  probably  shut. 

□  Pass 

□  Fail 

3.6  heck  emergency  stop 
valve.  Place  in  the 
run  position. 

Puts  stop  valve  in  run  position 

Correctly  places  stop  valve  in  run 
position. 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

Appendix  C  of  A  Method  for  Developing  Mariner  Assessments  Assessment  Procedures/Preparing  Main  Engine 


16  of  20 


Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


3.7  Start  engine  in  ahead 
direction. 

Report  that  engine  has  started, 
speed  and  direction 

Correct  action  ahead  and  correct 
report  of  speed  and  direction 

Incorrect  action  or  report  results  in 
automatic  failure  of  assessment. 

□  Pass 

□  Fail 

3.8  Check  lubricating 
pressure  upon 
starting  engine. 

Report  lubrication  oil  pressure 

Correctly  reports  lubrication  oil 
pressure 

Incorrect  report  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

3.9  Inspect  engine  while 
running. 

Proper  action  to  inspect  engine 

Correctly  inspects  engine  while 
running 

□  Pass 

□  Fail 

3.10  Stop  engine. 

Proper  action  to  stop  engine 

Correctly  stops  engine 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

3.11  Start  engine  in 
astern  direction. 

Report  that  engine  has  started, 
speed  and  direction 

Correctly  action  astern  and  correct 
report  of  speed  and  direction 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

3.12  Stop  engine. 

Proper  action  to  stop  engine 

Correctly  stops  engine 

Incorrect  action  results  in  automatic 
failure  of  assessment 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  □  Pass 

Total  the  number  of  “Pass"  scores  you  indicated  above.  Note  that  failure  of  3.1 ,  □  Fail 

3.6, 3.7, 3.8, 3.10, 3.1 1 ,  or  3.12  results  in  automatic  failure  of  this  assessment. 

Pass:  Score  of  9-12  Fail:  Score  of  0-8 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet.  _ 
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ASSESSMENT  WORKSHEET  IV 


Section  1.  Assessment  Conditions 


Assessment  Objective 

4.  Transfer  control  of  the  main  engine  to  a  remote  operating  station. 

Assessment  Method 

Shipboard  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards. 
The  assessor  should  tell  the  candidate  when  to  stop  the  engine  and  restart  in  an 
astern  direction. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  the  assessment  at  the  engineer's  local  control  station. 
During  the  accomplishment  of  this  task,  the  candidate  and  assessor  should  move 
to  the  remote  control  station. 

Initial  Condition 

The  main  engine  should  be  in  a  local  mode  of  operation  before  beginning  this 
assessment.  The  assessor  may  choose  to  place  engine  control  at  either  the 
remote  engineering  operating  station  or  the  bridge  station  (if  applicable). 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

4.1  Switch  control  of  engine 
from  local  console  to 
either  remote 
engineering  operating 
station  or  bridge. 

Verifies  engine  control 
transfer 

Correctly  transfers  control  to 
remote  station 

Incorrect  action  results  in  automatic 
failure  of  the  assessment. 

□  Pass 

□  Fail 

4.2  Start  engine  in  ahead 
direction. 

Reports  engine  direction 
and  speed 

Correctly  starts  engine  and  makes 
correct  report 

Incorrect  action  results  in  automatic 
failure  of  the  assessment. 

□  Pass 

□  Fail 

4.3  Start  engine  in  astern 
direction. 

Reports  engine  direction 
and  speed 

Correctly  starts  engine  and  makes 
correct  report 

Incorrect  action  results  in  automatic 
failure  of  the  assessment. 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  □  Pass 

Total  the  number  of  “Pass”  scores  you  indicated  above.  □  pail 

Pass:  Score  of  3  Fail:  Score  of  0-2 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet. 
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ASSESSMENT  WORKSHEET  V 


Section  1.  Assessment  Conditions 


Assessment  Objective 

5.  Prepare  main  propulsion  plant  to  answer  orders  from  the  bridge. 

Assessment  Method 

Shipboard  assessment. 

Candidate  Orientation 

Assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and  standards 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  and  assessor  should  begin  this  assessment  at  the  remote  operating 
station  (either  engine  room  or  bridge). 

Initial  Condition 

The  engine  should  be  in  a  remote  mode  of  operation  before  beginning  this 
assessment.  To  test  the  candidate's  knowledge  of  the  control  system,  the 
assessor  may  choose  to  change  settings  on  the  automatic  control. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

5.1  Open  clutch  control 
valve. 

Verifies  clutch  control  valve  is 
open 

Correctly  opens  clutch  control  valve 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

5.2  Ensure  proper  set  up 
of  automatic  controls. 

Verifies  proper  set  up  of 
automatic  controls 

Correctly  sets  up  automatic 
controls 

Incorrect  action  results  in  automatic 
failure  of  assessment. 

□  Pass 

□  Fail 

5.3  Advise  bridge  that  the 
propulsion  plant  is 
ready. 

Makes  proper  report  to  bridge 

Correctly  makes  report  to  bridge 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  q  Pass 

Total  the  number  of  “Pass”  scores  you  indicated  above.  □  Fail 

Pass:  Score  of  3  Fail:  Score  of  0-2 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet _ 
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Appendix  D  to 

“A  Method  for  Developing  Mariner  Assessments” 


MARINER  ASSESSMENT  PROCEDURES  FOR  LOCATING 
COMMON  FAULTS  AND  PREVENTING  DAMAGE  TO 

GENERATORS 


This  appendix  contains  example  procedures  for  assessing  a 
mariner’s  ability  to  locate  common  faults  and  prevent  damage  to 
generators.  The  assessment  package  consists  of  the  following: 
assessor  instructions,  candidate  instructions,  three  assessment 
worksheets,  and  an  assessment  control  sheet  for  documenting  the 
final  results  of  the  assessment. 


This  report  may  be  downloaded  from  the  U.S.  Coast  Guard  Research  and  Development  Center 

web  site  at  http://www.rdc.uscg.mil. 
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ASSESSOR  INSTRUCTIONS 


Introduction 

The  following  procedures  are  designed  to  evaluate  a  candidate’s  ability  to  determine  common 
faults  during  the  operation  of  a  ship  service  generator,  including  faults  that  occur  during  the  start 
and  the  operations.  The  candidate  must  identify  the  fault  and  take  corrective  action  to  maintain 
operations  or  reduce  the  risk  of  damage  to  the  generator  and  the  electrical  system. 

This  assessment  was  specifically  developed  for  an  electrical  generation  system  that  includes  at 
least  two  diesel  generators.  The  specific  system  used  in  the  generation  is  part  of  a  Norcontrol 
simulator  of  a  slow-speed  diesel  plant.  However,  this  assessment  may  be  applicable  to  any 
electrical  generation  plant  that  has  more  than  one  diesel  generator.  The  method  of  dealing  with 
faults  is  specifically  designed  for  a  simulator  and  may  not  be  directly  applicable  to  shipboard 
installations. 

The  assessment  materials  include  Assessor  Instructions,  Candidate  Instructions,  an  Assessment 
Control  Sheet,  and  three  separate  assessment  modules  ( Assessment  Worksheets  I-III).  The 
Assessor  Instructions  describe  the  assessment  objectives,  method,  performance  measures,  and 
performance  standards.  The  Assessor  Instructions  also  list  the  responsibilities  of  the  assessor 
during  each  phase  of  the  assessment  process.  The  Candidate  Instructions  explain  the  assessment 
process  from  the  candidate’s  perspective  and  define  the  responsibilities  of  the  candidate. 

The  Assessment  Control  Sheet  provides  a  record  of  the  names  of  the  candidate,  assessor,  and 
qualified  instructor  who  certified  that  the  candidate  met  the  training  prerequisites  for  assessment. 
The  control  sheet  also  includes  the  date  and  location  of  the  assessment  and  the  STCW  reference 
information  for  the  competency  area  being  assessed.  Section  3  of  the  Assessment  Control  Sheet 
provides  space  for  the  assessor  to  record  the  candidate’s  score  on  each  assessment  objective. 
Finally,  section  4  of  the  control  sheet  summarizes  the  assessment  objectives  and  the  actions 
comprising  each  objective. 

As  the  assessor,  you  will  evaluate  the  candidate’s  ability  to  complete  the  assessment  objectives 
correctly.  Using  the  scoring  procedures  described  below,  you  will  provide  the  candidate  with  a 
separate  score  for  each  assessment  objective  and  compile  these  scores  into  a  final  score  for  the 
entire  assessment.  The  scores  for  each  assessment  objective  and  the  candidate  s  overall  score  for 
the  entire  assessment  are  to  be  recorded  on  the  Assessment  Control  Sheet. 

The  rest  of  this  section  briefly  describes  the  various  components  of  this  assessment.  Assessment 
Worksheets  I  through  V  provide  additional  information  about  the  assessment  objectives,  method, 
conditions,  performance  measures,  performance  standards,  and  scoring  procedures. 

Assessment  Objectives 

The  candidate  should  evaluate  the  performance  of  the  generator  and  its  prime  mover,  identify 
faults  and  take  action  to  correct  the  fault  or  minimize  the  potential  damage.  You  should  assess 
the  candidate’s  ability  to: 

1 .  Identify  and  correct  faults  that  occur  during  engine  start. 
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2.  Identify  and  correct  operational  faults. 

3.  Restore  power  after  generator  failure. 

Assessment  Method 

These  objectives  should  be  assessed  in  an  engine  room  simulator  so  that  conditions  can  be  varied 
without  affecting  the  safety  of  a  vessel,  its  crew,  or  equipment.  The  simulator  can  be  either 
diesel  or  steam. 

Assessment  Conditions 

The  assessment  should  begin  at  the  local  control  station.  The  candidate  then  should  work 
throughout  the  engine  room  simulator.  As  the  assessor,  you  should  set  the  appropriate  faults  into 
the  simulator  controls  and  observe  the  candidate’s  actions.  The  candidate  should  not  receive  any 
assistance  from  others  during  the  assessment. 

Performance  Measures  and  Standards 

Identifying  and  correcting  faults  for  the  electrical  generators  involves  approximately  17  actions. 
These  actions  include  a  number  of  steps  that  must  be  completed  in  sequence  for  a  successful 
correction  of  the  problem.  The  performance  assessment  will  be  based  on  demonstrating  how  to 
analyze  the  problem  and  complete  each  step,  not  on  the  memorizing  the  exact  order  of  all  the 
actions. 

Use  pass/fail  scoring: 

□  Pass  -  Performance  demonstrates  an  ability  to  successfully  identify  the  fault  and  carry  out 
actions  to  correct  the  fault  or  prevent  damage  to  the  equipment. 

□  Fail  -  The  candidate  either  did  not  identify  the  fault,  or  when  identified,  did  not  take  proper 
corrective  action. 

Note  that  the  faults  in  assessment  objective  1  are  generally  not  as  critical  as  faults  in  objective  2, 
because  the  potential  for  damage  is  not  as  great.  In  objective  3,  if  the  candidate  doesn’t  perform 
an  action  properly,  he  or  she  will  not  restore  the  power.  Therefore,  most  of  the  items  in  objective 
3  are  required  items  -  that  is,  if  the  candidate  fails  these  items,  he  or  she  fails  the  entire 
assessment. 

Individual  measures  and  standards  for  each  assessment  objective  are  specified  on  Assessment 
Worksheets  I  through  III.  The  range  of  scores  acceptable  for  a  “Pass”  grade  is  indicated  at  the 
bottom  of  each  worksheet. 

Assessment  Checklist 

The  following  checklist  summarizes  the  tasks  and  responsibilities  of  the  assessor  at  each  phase  of 
the  assessment  process. 
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Preparing  for  the  Assessment 

□  Start  the  simulator  and  set  it  in  the  configuration  required  for  the  assessment. 

□  Set  desired  faults  in  the  simulator.  These  faults  are  in  the  objective  column  of  the  attached 
scoring  form.  One  fault  should  be  entered  for  each  of  the  objectives.  If  there  is  more  than 
one  possible  cause  of  the  fault,  the  assessor  may  set  any  one  of  the  possible  faults.  (The 
flexibility  of  the  simulator  may  determine  which  faults  can  be  set.)  There  are  no  faults  to  be 
set  for  the  third  task,  restoring  power  after  generator  failure.  The  assessor  must  set  up  the 
simulator  for  generator  failure. 

Briefing  the  Candidate  before  the  Assessment 

□  Provide  the  candidate  with  a  copy  of  the  Candidate  Instructions  and  Assessment  Control 
Sheet. 

□  Review  the  assessment  procedures  with  the  candidate  and  answer  any  questions. 

□  Explain  the  assessment  performance  measures  and  standards. 

□  Discuss  the  desired  outcome(s)  and  consequences  of  failing  to  perform  part  of  all  of  the 
assessment. 

□  Explain  that  the  assessment  will  be  terminated  if  the  candidate  does  not  correctly  perform  all 
of  the  “required”  actions  (i.e.,  those  with  an  italicized  note  in  the  “standards”  column  of  each 
Assessment  Worksheet). 

□  Remind  the  candidate  that  it  is  permissible  to  ask  questions  during  the  assessment,  especially 
if  he  or  she  has  a  safety  concern. 

□  Discuss  the  candidate’s  willingness  to  be  assessed  under  the  circumstances  presented. 

Observing  the  Candidate’s  Performance 

□  If  a  safety  violation  occurs,  terminate  the  assessment  immediately. 

□  Continuously  observe  the  candidate  during  the  assessment.  Require  that  standard  procedures 
or  company  policy  be  adhered  to  except  when  assessment  procedures  require  demonstration 
of  knowledge  or  skill  different  from  a  convention  adopted  by  the  company  or  facility. 

□  Ensure  realistic  assessment  conditions  consistent  with  a  normal  working  environment  for  a 
marine  engineer.  Ensure  the  candidate  can  concentrate  on  the  task  at  hand. 

□  Avoid  giving  the  candidate  unsolicited  assistance,  but  respond  to  appropriate  questions  and 
provide  appropriate  equipment  when  required. 

□  Remain  objective  and  maintain  positive  control  of  the  operation  at  all  times. 

Determining  Assessment  Outcome 

□  Record  performance  on  the  appropriate  assessment  worksheets. 

□  Apply  the  scoring  procedure. 
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□  Remember  that  if  the  candidate  has  incorrectly  performed  any  of  the  “required”  actions,  he  or 
she  automatically  fails  the  entire  assessment  (required  actions  are  noted  in  italics  in  the 
“standard”  column  of  each  assessment  worksheet). 

□  Determine  and  document  the  outcome  of  the  assessment,  then  transfer  the  final  results  to  the 
Assessment  Control  Sheet. 

Debriefing  the  Candidate 

□  Debrief  the  candidate  as  soon  as  possible  after  the  assessment. 

□  Provide  the  candidate  with  a  copy  of  the  Assessment  Control  Sheet. 

□  Restate  the  assessment  objectives. 

□  Focus  on  positive  accomplishments  first. 

□  Identify  areas  needing  improvement. 

□  If  the  candidate  failed  to  demonstrate  proficiency,  jointly  develop  an  improvement  plan  to 
prepare  for  reassessment. 
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Assessment  Procedures/Locating  Common  Faults 


CANDIDATE  INSTRUCTIONS 


In  this  assessment,  you  will  be  evaluated  on  your  ability  to  successfully  locate  common  faults 
and  prevent  damage  to  generators.  Table  1  shows  the  1995  STCW  Code  specification  for 
operating  generators  and  control  systems. 


Table  1.  STCW  Code  specification  for  operating  generators  and  control  systems. 


STCW  Requirement 

1995  STCW  Code,  Section  A-l  11/1  (p.  76):  Mandatory  minimum 
requirements  for  certification  of  ratings  for  officers  in  charge  of  an 
engineering  watch  in  a  manned  engine  room  or  designated  duty  engineers 
in  a  periodically  unmanned  engine  room. 

STCW  Function 

Electrical,  electronic  and  control  engineering  at  the  operational  level. 

STCW  Competence 

Operate  alternators,  generators  and  control  systems. 

STCW  Proficiency 

Generating  plant  -  Location  of  common  faults  and  action  to  prevent 
damage. 

Assessment  Objectives  and  Methods 

A  qualified  assessor  will  assess  you  on  your  ability  to  meet  the  three  assessment  objectives  in  the 
first  column  of  Table  2.  Note  that  each  assessment  objective  has  a  corresponding  set  of 
assessment  methods.  Performance  assessed  by  “practical  skill  demonstration”  means  your 
assessor  will  ask  you  to  demonstrate  your  ability  to  perform  the  actions  required  in  the  objective. 
Your  practical  skill  demonstration  will  occur  in  an  engine  room  simulator.  Performance  assessed 
by  “oral  test  questions”  means  your  assessor  will  ask  you  to  respond  to  one  or  more  questions 
and  then  assess  your  ability  to  respond  correctly  to  each  question. 


Table  2.  Assessment  objectives  and  methods  for  locating  common  faults  and 
preventing  damage  to  generators. 


Assessment  Objective 

Assessment  Method 

1 .  Identify  and  correct  faults  that  occur 
during  engine  start. 

Practical  skill  demonstration  in  an  engine  room 
simulator  /  Oral  test  questions 

2.  Identify  and  correct  operational 
faults. 

Practical  skill  demonstration  in  an  engine  room 
simulator  /  Oral  test  questions 

3.  Restore  power  after  generator  failure. 

Practical  skill  demonstration  in  an  engine  room 
simulator  /  Oral  test  questions 

Refer  to  the  Assessment  Control  Sheet  for  a  list  of  the  specific  actions  required  for  each  objective 
(you  should  receive  a  copy  of  this  document  from  the  assessor  during  your  pre-assessment 
briefing).  Below  are  some  general  guidelines  for  what  you  should  expect  during  the  assessment 
process. 
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Pre-Assessment  Briefing  with  Assessor 

This  briefing  should  occur  approximately  a  week  in  advance  of  the  scheduled  assessment,  if 
possible.  This  will  help  you  and  the  assessor  to  be  well  prepared  for  the  assessment  when  it 
actually  occurs.  During  this  briefing,  you  should: 

•  Discuss  your  prior  experience,  training,  and/or  company  policy  with  the  assessor.  On  the 
basis  of  these  qualifications,  discuss  whether  you  are  qualified  to  undertake  this 
assessment.  If  you  both  agree  you  are  qualified,  then  continue  with  the  assessment 
process.  If  not,  arrange  for  additional  on-the-job  or  simulator  training,  and  set  a  date  for 
another  review  of  your  qualifications. 

•  Obtain  a  copy  of  the  Assessment  Control  Sheet  from  your  assessor.  Review  this 
document  and  discuss  the  scope  and  depth  of  knowledge  covered  by  this  assessment. 

Ask  questions  about  any  part  of  the  assessment  that  is  unclear. 

•  Review  the  assessment  performance  measures  and  standards,  and  ask  any  questions  you 
have  about  them. 

•  Discuss  the  desired  outcome(s)  and  the  consequences  of  failing  to  perform  any  part  of  the 
assessment. 

•  Discuss  the  general  assessment  procedures.  Your  assessor  will  inform  you  of  how  much 
time  is  allowed,  when  the  assessment  begins,  and  under  what  circumstances  he  or  she 
will  terminate  the  assessment. 

•  Discuss  any  non-standard  procedures  that  are  expected  of  you  during  the  assessment. 

•  Consider  whether  you  are  willing  to  be  assessed  under  the  circumstances  presented  and 
advise  the  assessor  of  your  willingness  to  undertake  the  assessment. 

Participation  in  the  Assessment 

Your  assessment  will  begin  at  the  local  engineering  control  station.  During  the  assessment  you 
will  be  working  throughout  the  simulator. 

The  assessor  will  suspend  the  assessment  if  abnormal  conditions  develop  with  any  equipment  or 
if  the  safety  of  personnel  or  equipment  is  endangered.  He  or  she  will  also  suspend  the 
assessment  if  you  fail  to  correctly  perform  one  of  the  required  actions  for  this  assessment. 

Your  assessor  will  continuously  observe  you  during  the  assessment.  You  are  expected  to  adhere 
to  standard  procedures  or  company  policy  unless  the  assessor  briefs  you  on  a  requirement  to 
perform  a  non-standard  procedure.  During  the  assessment,  remember  to: 

•  Use  appropriate  marine  terminology  and  nomenclature  at  all  times. 

•  Listen  to  an  entire  question  before  responding  or  acting.  Remember  that  you  may  not  use 
reference  material  of  any  kind  during  the  assessment. 

•  Ask  questions  if  you  have  a  safety  concern.  The  assessor  will  provide  you  with 

appropriate  responses  to  your  questions.  If  a  safety  violation  occurs,  the  assessor  will 
terminate  the  assessment  immediately. 
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Your  assessor  will  ensure  realistic  assessment  conditions  consistent  with  the  normal  working 
environment  for  an  engineer. 

The  Outcome  of  Your  Assessment 

The  assessor  will  record  your  performance  on  a  series  of  Assessment  Worksheets.  He  or  she  will 
score  each  assessment  objective  on  a  “Pass/Fail”  basis.  Acceptable  performance  (a  “Pass”  score) 
will  be  based  on  your  ability  to  correctly  respond  to  test  questions.  It  will  also  be  based  on  your 
ability  to  perform  assigned  tasks  safely  in  a  manner  that  demonstrates: 

•  The  required  level  of  skill,  knowledge,  and  ability. 

•  Sound  and  professional  judgment. 

Unacceptable  performance  (a  “Fail”  score)  will  be  based  on  your  failure  to  perform  a  critical 
phase  of  the  assessment  exercise  proficiently.  When  this  occurs,  your  assessment  will  be 
suspended  and  postponed  until  you  receive  further  instruction  and  training. 

Note  that  the  following  events  will  terminate  an  assessment  immediately: 

•  An  action,  or  lack  of  action,  by  you  which  required  corrective  action  or  intervention  by 
the  assessor  to  prevent  injury,  damage,  or  the  development  of  a  hazardous  condition. 

•  Your  failure  to  use  proper  procedures,  including  appropriate  communication  procedures, 
during  the  assessment. 

•  Your  failure  to  take  prompt  corrective  action  when  required. 

The  assessor  will  strictly  adhere  to  pre-determined  scoring  procedures,  and  will  document  the 
outcome  of  the  assessment.  He  or  she  will  transfer  the  final  results  of  your  assessment  to  the 
Assessment  Control  Sheet. 

Assessment  Debriefing 

You  and  the  assessor  should  discuss  the  assessment  results  as  soon  as  possible  after  the 
assessment.  During  the  debriefing: 

•  The  assessor  should  provide  you  with  a  copy  of  the  Assessment  Control  Sheet  describing 
the  results  of  your  assessment. 

•  The  assessor  should  restate  the  assessment  objective(s)  and  identify  those  that  you 
successfully  demonstrated. 

•  If  appropriate,  you  and  the  assessor  should  discuss  the  areas  in  which  you  need 
improvement,  and  then  develop  an  improvement  plan  based  on  the  assessment  outcome. 
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ASSESSMENT  CONTROL  SHEET 


Section  1.  Assessment  Reference  Information 


1.  Name  of  Candidate 

2.  Name  of  Designated  Assessor 

3.  Signature  of  Qualified  Instructor1 

4.  Date  of  Assessment 

5.  Assessment  Location2 

Section  2.  STCW  Reference  Information 


Assessment  Area 

Locating  common  faults  and  preventing  damage  to  generators. 

Assessment  Method 

Simulator  assessment  and  oral  test  questions. 

STCW  Requirement 

STCW  Code,  Section  A-lll/1 ,  p.  76  -  Mandatory  minimum  requirements  for  certification 
of  ratings  for  officers  in  charge  of  an  engineering  watch  in  a  manned  engine  room  or 
designated  duty  engineers  in  a  periodically  unmanned  engine  room. 

STCW  Function 

Electrical,  electronic  and  control  engineering  at  the  operational  level. 

STCW  Competence 

Operate  alternators,  generators  and  control  systems. 

STCW  Proficiency 

Generating  plant  -  Location  of  common  faults  and  action  to  prevent  damage. 

Section  3.  Assessment  Objectives,  Methods,  and  Scores 


Assessment  Objective 

Assessment  Method 

Score 

1 .  Identify  and  correct  faults  that 
occur  during  engine  start. 

Practical  skill  demonstration  in  an  engine 
room  simulator/  Oral  test  questions 

□  Pass  □  Fail 

2.  Identify  and  correct  operational 
faults. 

Practical  skill  demonstration  in  an  engine 
room  simulator/  Oral  test  questions 

□  Pass  □  Fail 

3.  Restore  power  after  generator 
failure. 

Practical  skill  demonstration  in  an  engine 
room  simulator/  Oral  test  questions 

□  Pass  □  Fail 

COMMENTS: 

FINAL  SCORE 

□  Pass  □  Fail 

1  The  qualified  instructor  certifies  that  the  candidate  has  met  the  training  prerequisites. 

2  Write  the  name  of  the  simulator  or  vessel  used  during  the  assessment  (e.g.,  Massachusetts 
Maritime  Academy  training  simulator,  ARCO  California,  etc.). 
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Section  4.  Assessment  Worksheet  Summary 


Assessment 

Worksheet 

Assessment  Objective 

Action 

1 

1 .  Identify  and  correct  faults 
that  occur  during  engine 
start. 

1 .1  Identify  and  correct  fault  of  lack  of  pump  oil 
pump  pressure. 

1 .2  Identify  and  correct  lack  of  starting  air. 

1 .3  Identify  and  correct  lack  of  fuel  oil  pressure. 

1 .4  Identify  and  correct  failure  to  start  due  to 
overspeed  trip. 

1 .5  Identify  and  correct  failure  of  prime  mover  to 
come  up  to  speed. 

1 .6  Identify  and  correct  failure  of  generator  to 
synchronize  with  the  bus. 

II 

2.  Identify  and  correct 
operational  faults. 

2.1  Identify  and  correct  loss  of  lube  oil  pressure. 

2.2  Identify  and  correct  overheating  generator. 

2.3  Identify  and  correct  overheating  of  prime  mover. 

2.4  Identify  and  correct  overload  of  prime  mover. 

2.5  Identify  and  correct  loss  of  fuel  oil  pressure. 

2.6  Identify  and  correct  low  cycles. 

2.7  Identify  and  correct  for  overspeed. 

III 

3.  Restore  power  after 
generator  failure 

3.1  Ensure  non-operationa!  engine  is  not  connected 
to  bus. 

3.2  Start  idle  engine  and  generator. 

3.3  Connect  generator  to  bus. 

3.4  Secure  emergency  generator. 
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ASSESSMENT  WORKSHEET  I 


Section  1.  Assessment  Conditions 


Assessment  Objective 

1.  Identify  and  correct  faults  that  occur  during  engine  start. 

Assessment  Method 

Engine  room  simulator  assessment  and  oral  test  questions. 

Candidate  Orientation 

The  candidate  should  have  a  strong  familiarization  with  the  equipment  involved  and 
the  operations  of  the  simulator.  The  assessor  will  be  either  observing  the  candidate 
directly  or  from  the  simulator  control  station. 

The  candidate  should  attempt  to  start  the  generator.  The  candidate  should  then 
identify  and  correct  the  faults  the  assessor  has  programmed.  The  assessor  will  set 
the  faults  into  the  simulator  and  tell  the  candidate  when  to  start.  The  assessor  will 
then  check  to  be  sure  that  the  task  is  properly  completed. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  the  assessment  at  the  engineer’s  local  operating  console 
in  the  simulator. 

Initial  Conditions 

The  auxiliary  generator  and  its  prime  mover  are  secured  but  operational  and 
available  for  use.  All  the  support  systems  are  lined  up  and  operational.  The  area  is 
clear  and  the  assessor  can  observe  activities.  The  assessor  should  create  the 
following  faults;  however,  he  or  she  can  change  the  order  in  which  they  occur: 

□  Prime  mover  fails  to  start  due  to: 

•  Lack  of  lube  oil  pump  of  pressure. 

•  Lack  of  starting  air  pressure. 

•  Lack  of  fuel  oil  pressure. 

•  Failure  to  reset  the  overspeed  trip. 

□  Prime  mover  will  not  come  up  to  speed. 

□  Generator  will  not  synchronize  with  electrical  bus. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

1.1  Identify  and  correct 
fault  of  lack  of  oil 
pump  pressure. 

Checks  and  corrects  lube  oil 
level 

Starts  lube  oil  pump 

Correctly  checks  and  corrects 
lube  oil  level 

Correctly  starts  lube  oil  pump 

□  Pass 

□  Fail 

1.2  Identify  and  correct 
lack  of  starting  air. 

Checks  and  corrects  starting  air 
valve 

Checks  and  corrects  starting  air 
compressor 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks  and  corrects: 

□  Air  valve. 

□  Air  compressor. 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

Appendix  D  of  A  Method  for  Developing  Mariner  Assessments  Assessment  Procedures/Locating  Common  Faults 


10  of  15 


Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


1 .3  Identify  and  correct 
lack  of  fuel  oil 
pressure. 

Checks  and  corrects  fuel  oil  line 
up 

Checks  and  corrects  dirty  fuel 
oil  filters/strainers 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks  and  corrects: 

□  Fuel  oil  line  up. 

□  Dirty  fuel  oil  filters/strainers. 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

1.4  Identify  and  correct 
failure  to  start  due  to 
overspeed  trip. 

Check  and  reset  overspeed  trip 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks  and  resets 
overspeed  trip 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

1.5  Identify  and  correct 
failure  of  prime 
mover  to  come  up  to 
speed 

Checks  and  corrects  low  fuel  oil 
pressure 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks  and  corrects 
low  fuel  oil  pressure 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

1.6  Identify  and  correct 
failure  of  generator 
to  synchronize  with 
the  bus 

Checks  and  corrects  low 
voltage 

Checks  and  corrects  low  cycles 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks  and  corrects: 

□  Low  voltage. 

□  Low  cycles. 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  q  Pass 

Total  the  number  of  “Pass”  scores  you  indicated  above.  Q  paj| 

Pass:  Score  of  5-6  Fail:  Score  of  0-4 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet. 
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ASSESSMENT  WORKSHEET  II 


Section  1.  Assessment  Conditions 


Assessment  Objectives 

2.  Identify  and  correct  operational  faults. 

Assessment  Method 

Engine  room  simulator  assessment  and  oral  test  questions. 

Candidate  Orientation 

The  assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and 
standards  and  then  either  observes  the  candidate  directly  or  from  the  simulator 
control  station.  The  candidate  should  demonstrate  the  ability  to  identify  and  correct 
faults  that  occur  on  an  operating  generator.  This  demonstration  of  knowledge  will 
be  verified  by  observing  the  candidate  reacting  to  the  faults  the  assessor  has 
programmed. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  this  assessment  from  the  electrical  control  station. 
During  the  assessment,  the  candidate  may  need  to  return  to  the  engineer’s  local 
operating  stations  in  the  simulator. 

Initial  Conditions 

The  prime  mover  and  generator  should  be  operating  properly  and  should  be 
connected  to  the  electrical  bus.  It  may  be  necessary  for  the  candidate  to  operate  a 
second  generator  to  correct  the  faults.  To  create  faults,  the  assessor  should  vary 
the  following  items.  He  or  she  may  vary  the  order  in  which  these  faults  occur: 

□  Loss  of  lube  oil  pressure. 

□  Overheating  of  the  generator. 

□  Overheating  of  the  prime  mover. 

□  Overload  of  the  prime  mover. 

□  Loss  of  fuel  oil  pressure  for  the  prime  mover. 

□  Generator  cycles  below  operating  level. 

□  Overspeed  of  prime  mover. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

2.3  Identify  and  correct 
loss  of  lube  oil 
pressure. 

Checks  engine  for  lube  oil 
leaks 

Stops  engine,  if  required 

Starts  engine,  if  required 

Correctly  checks  engine  for  lube  oil 
leaks 

Correctly  starts  or  stops  engine,  as 
required 

□  Pass 

□  Fail 

2.2  Identify  and  correct 
overheating 
generator. 

Checks  load  on  generator 

Reduces  load  on  generator 

Checks  fresh  water  cooling 

Checks  ventilation  level 

Correctly  checks  and  reduces  load 
on  generator 

Correctly  checks: 

□  Fresh  water  cooling. 

□  Ventilation  level. 

□  Pass 

□  Fail 

Appendix  D  of  A  Method for  Developing  Mariner  Assessments  Assessment  Procedures/Locating  Common  Faults 


Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores  (continued) 


2.3  Identify  and  correct 
overheating  of  prime 
mover. 

Checks  fresh  water-cooling  to 
engine 

Checks  ventilation  in  engine 
room 

Answers  questions: 

What  are  the  potential  causes 
of  the  fault? 

How  should  the  equipment 
respond  to  your  actions? 

Correctly  checks: 

□  Fresh  water-cooling  to  engine. 

□  Ventilation  in  engine  room. 

Correctly  responds  to  both 
questions 

□  Pass 

□  Fail 

2.4  Identify  and  correct 
overload  of  prime 
mover. 

Checks  electrical  load 

Reduces  load  as  required 

Correctly  checks  electrical  load  of 
prime  mover 

Correctly  reduces  electrical  load  of 
prime  mover 

□  Pass 

□  Fail 

2.5  Identify  and  correct 
loss  of  fuel  oil 
pressure. 

Checks  filters/strainers 

Checks  fuel  oil  pump 

Correctly  checks: 

□  Filters/strainers. 

□  Fuel  oil  pump. 

□  Pass 

□  Fail 

2.6  Identify  and  correct 
low  cycles. 

Checks  for  overload 

Checks  engine  speed 

Correctly  checks: 

□  For  overload. 

□  Engine  speed. 

□  Pass 

□  Fail 

2.7  Identify  and  correct 
for  overspeed. 

Starts  idle  engine  and  places 
online 

Checks  engine  for  cause  of 
overspeed 

Correctly  starts  engine  and  places 
it  online 

Correctly  checks  engine  for  cause 
of  overspeed 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  □  Pass 

Total  the  number  of  “Pass"  scores  you  indicated  above.  □  Fail 

Pass:  Score  of  6-7  Fail:  Score  of  0-5 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet. 
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ASSESSMENT  WORKSHEET  III 


Section  1.  Assessment  Conditions 


Assessment  Objective 

3.  Restore  power  after  generator  fails. 

Assessment  Method 

Engine  room  simulator  assessment  and  oral  test  questions. 

Candidate  Orientation 

The  assessor  briefs  the  candidate  on  assessment  methods,  conditions,  and 
standards.  The  candidate  should  demonstrate  the  ability  to  start  the  idle  engine 
and  generator.  To  successfully  meet  this  objective,  the  candidate  must  complete 
the  following  assessment  objectives,  which  must  be  completed  in  the  proper  order. 

Required  Equipment, 
Apparatus,  and/or  Tools 

The  candidate  should  begin  the  assessment  at  the  engineer’s  local  control  station. 

Initial  Condition 

The  assessor  should  ensure  that  the  emergency  generator  is  operating.  One 
prime  mover  and  generator  should  have  no  faults  and  should  be  ready  to  start. 

The  generator  supplying  the  electrical  bus  fails  and  the  emergency  generator 
starts.  One  generator  and  its  prime  mover  are  ready  to  start.  There  is  one 
candidate  starting  the  idle  engine. 

Section  2.  Actions,  Performance  Measures,  Standards,  and  Scores 


Action 

Performance  Measure 

Performance  Standard 

Score 

3.1  Ensure  non- 

operational  engine  is 
not  connected  to  bus. 

Checks  the  bus  is  non- 
operational 

Correctly  checks  the  bus 

□  Pass 

□  Fail 

3.2  Start  idle  engine  and 
generator. 

Starts  idle  engine  and 
generator 

Correctly  carries  out  start 
procedures 

□  Pass 

□  Fail 

3.3  Connect  generator  to 
bus. 

Connects  generator  to  bus 

Correctly  carries  out  procedures  for 
connecting  generator  to  bus 

□  Pass 

□  Fail 

3.4  Secure  emergency 
generator. 

Ensures  generator  is  operating 
properly 

!  Secures  emergency  generator 

Answers  question: 

What  electrical  loads  does  the 
emergency  generator  carry? 

Correctly  ensures  generator  is 
operating  Correctly  secures 
emergency  generator 

Correctly  answers  question 

□  Pass 

□  Fail 

SCORING  PROCEDURE  FINAL  SCORE  □  Pass 

Total  the  number  of  ‘Pass’  scores  you  indicated  above.  □  paj| 

Pass:  Score  of  3-4  Fail:  Score  of  0-2 

Transfer  the  final  score  at  right  to  the  Assessment  Control  Sheet. 
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